Parallel Computing for Data Science

Transcription

1 Parallel Computing for Data Science With Examples in R, C++ and CUDA Norman Matloff University of California, Davis USA (g) CRC Press Taylor & Francis Group Boca Raton London New York CRC Press is an imprint of the Taylor St Francis Croup, an informa business A CHAPMAN & HALL BOOK

2 Contents Preface xix Bio xxiii 1 Introduction to Parallel Processing in R Recurring Theme: The Principle of Pretty Good Parallelism Fast Enough "R+X" A Note on Machines Recurring Theme: Hedging One's Bets Extended Example: Mutual Web Outlinks Serial Code Choice of Parallel Tool Meaning of "snow" in This Book Introduction to snow Mutual Outlinks Problem, Solution Code Timings Analysis of the Code 11 vii

3 viii CONTENTS 2 "Why Is My Program So Slow?": Obstacles to Speed Obstacles to Speed Performance and Hardware Structures Memory Basics Caches Virtual Memory Monitoring Cache Misses and Page Faults Locality of Reference Network Basics Latency and Bandwidth Two Representative Hardware Platforms: Multicore Machines and Clusters Multicore Clusters The Principle of "Just Leave It There" Thread Scheduling How Many Processes/Threads? Example: Mutual Outlink Problem "Big O" Notation Data Serialization "Embarrassingly Parallel" Applications What People Mean by "Embarrassingly Parallel" Suitable Platforms for Non-Embarrassingly Parallel Applications 34 3 Principles of Parallel Loop Scheduling General Notions of Loop Scheduling Chunking in snow 38

4 CONTENTS ix Example: Mutual Outlinks Problem A Note on Code Complexity Example: All Possible Regressions Parallelization Strategies The Code Sample Run Code Analysis Our Task List Chunking Task Scheduling The Actual Dispatching of Work Wrapping Up Timing Experiments The partools Package Example: All Possible Regressions, Improved Version Code Code Analysis Timings Introducing Another Tool: multicore Source of the Performance Advantage Example: All Possible Regressions, Using multicore Issues with Chunk Size Example: Parallel Distance Computation The Code Timings The foreach Package Example: Mutual Outlinks Problem 70

5 X CONTENTS A Caution When Using foreach Stride Another Scheduling Approach: Random Task Permutation The Math The Random Method vs. Others, in Practice Debugging snow and multicore Code Debugging in snow Debugging in multicore 78 4 The Shared-Memory Paradigm: A Gentle Introduction via R So, What Is Actually Shared? Global Variables Local Variables: Stack Structures Non-Shared Memory Systems Clarity of Shared-Memory Code High-Level Introduction to Shared-Memory Programming: Rdsm Package Use of Shared Memory Example: Matrix Multiplication The Code Analysis The Code A Closer Look at the Shared Nature of Our Data Timing Comparison Leveraging R Shared Memory Can Bring A Performance Advantage Locks and Barriers 94

6 CONTENTS xi Race Conditions and Critical Sections Locks Barriers Example: Maximal Burst in a Time Series The Code Example: Transforming an Adjacency Matrix The Code Overallocation of Memory Timing Experiment Example: k-means Clustering The Code Timing Experiment The Shared-Memory Paradigm: C Level OpenMP Example: Finding the Maximal Burst in a Time Series The Code Compiling and Running Analysis A Cautionary Note About Thread Scheduling Setting the Number of Threads Timings OpenMP Loop Scheduling Options OpenMP Scheduling Options Scheduling through Work Stealing Example: Transforming an Adjacency Matrix The Code 127

7 5.4.2 Analysis of the Code Example: Adjacency Matrix, R-Callable Code The Code, for.c() Compiling and Running Analysis The Code, for Repp Compiling and Running Code Analysis Advanced Repp Speedup in C Run Time vs. Development Time Further Cache/Virtual Memory Issues Reduction Operations in OpenMP Example: Mutual In-Links The Code Sample Run Analysis Cache Issues Rows vs. Columns Processor Affinity Debugging Threads Commands in GDB Using GDB on C/C++ Code Called from R Intel Thread Building Blocks (TBB) Lockfree Synchronization 155

8 CONTENTS xni 6 The Shared-Memory Paradigm: GPUs Overview Another Note on Code Complexity Goal of This Chapter Introduction to NVIDIA GPUs and CUDA Example: Calculate Row Sums NVIDIA GPU Hardware Structure Cores Threads The Problem of Thread Divergence "OS in Hardware" Grid Configuration Choices Latency Hiding in GPUs Shared Memory More Hardware Details Resource Limitations Example: Mutual Inlinks Problem The Code Timing Experiments Synchronization on GPUs Data in Global Memory Is Persistent R and GPUs Example: Parallel Distance Computation The Intel Xeon Phi Chip Thrust and Rth Hedging One's Bets 179

9 xiv CONTENTS 7.2 Thrust Overview Rth Skipping the C Example: Finding Quantiles The Code Compilation and Timings Code Analysis Introduction to Rth The Message Passing Paradigm Message Passing Overview The Cluster Model Performance Issues Rmpi Installation and Execution Example: Pipelined Method for Finding Primes Algorithm The Code Timing Example Latency, Bandwdith and Parallelism Possible Improvements Analysis of the Code Memory Allocation Issues Message-Passing Performance Subtleties Blocking vs. Nonblocking I/O The Dreaded Deadlock Problem 205

10 CONTENTS xv 9 MapReduce Computation Apache Hadoop Hadoop Streaming Example: Word Count Running the Code Analysis of the Code Role of Disk Files Other MapReduce Systems R Interfaces to MapReduce Systems An Alternative: "Snowdoop" Example: Snowdoop Word Count Example: Snowdoop k-means Clustering Parallel Sorting and Merging The Elusive Goal of Optimality Sorting Algorithms Compare-and-Exchange Operations Some "Representative" Sorting Algorithms Example: Bucket Sort in R Example: Quicksort in OpenMP Sorting in Rth Some Timing Comparisons Sorting on Distributed Data Hyperquicksort Parallel Prefix Scan General Formulation Applications General Strategies 235

11 xvi CONTENTS A Log-Based Method Another Way Implementations of Parallel Prefix Scan Parallel cumsum() with OpenMP Stack Size Limitations Let's Try It Out Example: Moving Average Rth Code Algorithm Performance Use of Lambda Functions Parallel Matrix Operations Tiled Matrices Example: Snowdoop Approach Parallel Matrix Multiplication Multiplication on Message-Passing Systems Distributed Storage Fox's Algorithm Overhead Issues Multiplication on Multicore Machines Overhead Issues Matrix Multiplication on GPUs Overhead Issues BLAS Libraries Overview Example: Performance of OpenBLAS 261

12 CONTENTS xvii 12.6 Example: Graph Connectedness Analysis The "Log Trick" Parallel Computation The matpow Package Features Solving Systems of Linear Equations The Classical Approach: Gaussian Elimination and the LU Decomposition The Jacobi Algorithm Parallelization Example: R/gputools Implementation of Jacobi QR Decomposition Some Timing Results Sparse Matrices Inherently Statistical Approaches: Subset Methods Chunk Averaging Asymptotic Equivalence O(-) Analysis Code Timing Experiments Example: Quantile Regression Example: Logistic Model Example: Estimating Hazard Functions Non-i.i.d. Settings Bag of Little Bootstraps Subsetting Variables 283

13 XVlll CONTENTS A Review of Matrix Algebra 285 A.l Terminology and Notation 285 A.1.1 Matrix Addition and Multiplication 286 A.2 Matrix Transpose 287 A.3 Linear Independence 288 A.4 Determinants 288 A.5 Matrix Inverse 288 A.6 Eigenvalues and Eigenvectors 289 A.7 Matrix Algebra in R 290 B R Quick Start 293 B.l Correspondences 293 B.2 Starting R 294 B.3 First Sample Programming Session 294 B.4 Second Sample Programming Session 298 B.5 Third Sample Programming Session 300 B.6 The R List Type 301 B.6.1 The Basics 301 B.6.2 The Reduce() Function 302 B.6.3 S3 Classes 302 B.6.4 Handy Utilities 304 B.7 Debugging in R 305 C Introduction to C for R Programmers 307 C.0.1 Sample Program 307 CO.2 Analysis 308 C.l C Index 311