Introducing R: From Your Laptop to HPC and Big Data
|
|
|
- Todd Ryan
- 10 years ago
- Views:
Transcription
1 Introducing R: From Your Laptop to HPC and Big Data Drew Schmidt and George Ostrouchov November 18, pbdr Core Team Introduction to pbdr
2 The pbdr Core Team Wei-Chen Chen 1 George Ostrouchov 1,2 Pragneshkumar Patel 2 Drew Schmidt 2 Support 112,896 Cores in 18,816 Nodes with 147 TB Memory This work used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR This work also used resources of National Institute for Computational Sciences at the University of Tennessee, Knoxville, which is supported by the Office of Cyberinfrastructure of the U.S. National Science Foundation under Award No. ARRA-NSF-OCI for NICS-RDAV center. This work used resources of the Newton HPC Program at the University of Tennessee, Knoxville. 1 Oak Ridge National Laboratory. Supported in part by the project Visual Data Exploration and Analysis of Ultra-large Climate Data funded by U.S. DOE Office of Science under Contract No. DE-AC05-00OR University of Tennessee. Supported in part by the project NICS Remote Data Analysis and Visualization Center funded by the Office of Cyberinfrastructure of the U.S. National Science Foundation under Award No. ARRA-NSF-OCI for NICS-RDAV center. pbdr Core Team Introduction to pbdr
3 About This Presentation Downloads This presentation and supplemental materials are available at: Sample R scripts and pbs job scripts available on Nautilus from: /lustre/medusa/proj/pbdr/tutorial/scripts.tar.gz pbdr Core Team Introduction to pbdr
4 About This Presentation Speaking Serial R with a Parallel Accent The content of this presentation is based in part on the pbddemo vignette Speaking Serial R with a Parallel Accent It contains more examples, and sometimes added detail. pbdr Core Team Introduction to pbdr
5 About This Presentation Installation Instructions Installation instructions for setting up a pbdr environment are available: This includes instructions for installing R, MPI, and pbdr. pbdr Core Team Introduction to pbdr
6 About This Presentation Conventions For Code Presentation We will use two different forms of syntax highlighting. One for displaying results from an interactive R session: 1 R> " interactive " 2 [1] " interactive " and one for presenting R scripts 1 " not interactive " pbdr Core Team Introduction to pbdr
7 Contents 1 Introduction to R 2 pbdr 3 Introduction to pbdmpi 4 The Generalized Block Distribution 5 Basic Statistics Examples 6 Introduction to pbddmat and the DMAT Structure 7 Examples Using pbddmat 8 Wrapup pbdr Core Team Introduction to pbdr
8 Contents 1 Introduction to R What is R? Basic Numerical Operations in R R Syntax for Data Science: Not A Matlab Clone! pbdr Core Team Introduction to pbdr
9 What is R? What is R? lingua franca for data analytics and statistical computing. Part programming language, part data analysis package. Dialect of S (Bell Labs). Syntax designed for data. pbdr Core Team Introduction to pbdr 1 / 99
10 What is R? Who uses R? Google, Pfizer, Merck, Bank of America, Shell a, Oracle b, Facebook, bing, Mozilla, okcupid c, ebay d, kickstarter e, the New York Times f a business-computing/07program.html?_r=0 b features-oracle-r-enterprise html c companies-using-r.php d using-r-in-production-industry-experts-share-their-experiences. html e kickstarter-facilitates-50m-in-indie-game-funding.html f nyt-charts-the-facebook-ipo-with-r.html pbdr Core Team Introduction to pbdr 2 / 99
11 What is R? Language Paradigms Functional Haskell BASH R Dynamic Java Fortran 2003 OO pbdr Core Team Introduction to pbdr 3 / 99
12 What is R? Data Types Storage: logical, int, double, double complex, character Structures: vector, matrix, array, list, dataframe Caveats: (Logical) TRUE, FALSE, NA For the remainder of the tutorial, we will restrict ourselves to real number matrix computations. pbdr Core Team Introduction to pbdr 4 / 99
13 What is R? Basics (1 of 2) The default method is to print: 1 R> sum 2 function (..., na.rm = FALSE ). Primitive (" sum ") Use <- for assignment: 1 R> x <- 1 2 R> x+1 3 [1] 2 Naming rules: mostly like C. R is case sensitive. We use. the way most languages use _, e.g., La.svd() instead of La_svd(). We use $ the way most languages use. pbdr Core Team Introduction to pbdr 5 / 99
14 What is R? Basics (2 of 2) Use? or?? to search help 1 R>? set. seed 2 R>? comm. set. seed 3 No documentation for c o m m. set. s e e d in specified packages and libraries : 4 you could try?? comm. set. s e e d 5 R>?? comm. set. seed pbdr Core Team Introduction to pbdr 6 / 99
15 What is R? Addons and Extras R has the Comprehensive R Archive Network (CRAN), which is a package repository like CTAN and CPAN. From R 1 install. packages (" pbdmpi ") # install 2 library ( pbdmpi ) # load From Shell 1 R CMD INSTALL pbdmpi _ tar. gz pbdr Core Team Introduction to pbdr 7 / 99
16 Basic Numerical Operations in R Lists (1 of 1) 1 R> l <- list (a=1, b="a") 2 R> l 3 $a 4 [1] $b 7 [1] "a" 8 9 R> l$a 10 [1] R> list (x= list (a=1, b="a"), y= TRUE ) 13 $x 14 $x$a 15 [1] $x$b 18 [1] "a" $y 22 [1] TRUE pbdr Core Team Introduction to pbdr 8 / 99
![(a=1, b="a"), y= TRUE ) 13 $x 14 $x$a 15 [1] 1 16 17 $x$b 18 [1] "a" 19 20 21](/docs-images/41/2793906/images/page_16.jpg "$y 22 [1] TRUE http://r-pbd.")
17 Basic Numerical Operations in R Vectors and Matrices (1 of 2) 1 R> c(1, 2, 3, 4, 5, 6) 2 [1] R> matrix (1:6, nrow =2, ncol =3) 5 [,1] [,2] [,3] 6 [1,] [2,] R> x <- matrix (1:6, nrow =2, ncol =3) 11 R> x[, -1] 12 [,1] [,2] 13 [1,] [2,] R> x[1, 1:2] 17 [1] pbdr Core Team Introduction to pbdr 9 / 99
![9 R> x <- matrix (1:6, nrow =2, ncol =3) 11 R> x[, -1] 12 [,1] [,2] 13 [1,] 3 5 14 [2,] 4 6 15](/docs-images/41/2793906/images/page_17.jpg "16 R> x[1, 1:2] 17 [1] 1 3 http://r-pbd.org/tutorial pbdr Core Team Introduction to pbdr 9 / 99")
18 Basic Numerical Operations in R Vectors and Matrices (2 of 2) 1 R> dim (x) 2 [1] R> dim ( x) <- NULL 5 R> x 6 [1] R> dim (x) <- c (3,2) 9 R> x 10 [,1] [,2] 11 [1,] [2,] [3,] pbdr Core Team Introduction to pbdr 10 / 99
![dim (x) <- c (3,2) 9 R> x 10 [,1] [,2] 11 [1,] 1 4 12 [2,] 2 5 13 [3,]](/docs-images/41/2793906/images/page_18.jpg "3 6 http://r-pbd.")
19 Basic Numerical Operations in R Vector and Matrix Arithmetic (1 of 2) 1 R> 1:4 + 4:1 2 [1] R> x <- matrix (0, nrow =2, ncol =3) 5 6 R> x [,1] [,2] [,3] 8 [1,] [2,] R> x + 1:3 12 [,1] [,2] [,3] 13 [1,] [2,] pbdr Core Team Introduction to pbdr 11 / 99
![[,2] [,3] 8 [1,] 1 1 1 9 [2,] 1 1 1 10 11 R> x + 1:3 12 [,1] [,2] [,3] 13 [1,] 1](/docs-images/41/2793906/images/page_19.jpg "3 2 14 [2,] 2 1 3 http://r-pbd.")
20 Basic Numerical Operations in R Vector and Matrix Arithmetic (2 of 2) 1 R> x <- matrix (1:6, nrow =2) 2 3 R> x*x 4 [,1] [,2] [,3] 5 [1,] [2,] R> x %*% x 9 Error in x %*% x : non - conformable arguments R> t(x) %*% x 12 [,1] [,2] [,3] 13 [1,] [2,] [3,] R> crossprod (x) 18 [,1] [,2] [,3] 19 [1,] [2,] [3,] pbdr Core Team Introduction to pbdr 12 / 99
![t(x) %*% x 12 [,1] [,2] [,3] 13 [1,] 5 11 17 14 [2,] 11 25 39 15 [3,] 17 39 61 16 17 R> crossprod (x) 18 [,1] [,2]](/docs-images/41/2793906/images/page_20.jpg "[,3] 19 [1,] 5 11 17 20 [2,] 11 25 39 21 [3,] 17 39 61 http://r-pbd.")
21 Basic Numerical Operations in R Linear Algebra (1 of 2): Matrix Inverse x n n invertible y n n (xy = yx = Id n n ) 1 R> x <- matrix ( rnorm (5* 5), nrow =5) 2 R> y <- solve ( x) 3 4 R> round (x %*% y) 5 [,1] [,2] [,3] [,4] [,5] 6 [1,] [2,] [3,] [4,] [5,] pbdr Core Team Introduction to pbdr 13 / 99
22 Basic Numerical Operations in R Linear Algebra (2 of 2): Singular Value Decomposition x = UΣV T 1 R> x <- matrix ( rnorm (2* 3), nrow =3) 2 R> svd (x) 3 $d 4 [1] $u 7 [,1] [,2] 8 [1,] [2,] [3,] $v 13 [,1] [,2] 14 [1,] [2,] pbdr Core Team Introduction to pbdr 14 / 99
23 R Syntax for Data Science: Not A Matlab Clone! More than just a Matlab clone... Data science (machine learning, statistics, data mining,... ) is mostly matrix algebra. So what about Matlab/Python/Julia/...? The one you prefer depends more on your religion rather than differences in capabilities. As a data analysis package, R is king. pbdr Core Team Introduction to pbdr 15 / 99
24 R Syntax for Data Science: Not A Matlab Clone! Simple Statistics (1 of 2): Summary Statistics 1 R> x <- matrix ( rnorm (30, mean =10, sd =3), nrow =10) 2 3 R> mean (x) 4 [1] R> median ( x) 7 [1] R> sd(as. vector (x)) 10 [1] R> colmeans ( x) 13 [1] R> apply (x, MARGIN =2, FUN =sd) 16 [1] pbdr Core Team Introduction to pbdr 16 / 99
25 R Syntax for Data Science: Not A Matlab Clone! Simple Statistics (2 of 2): Sample Covariance cov(x n p ) = 1 n 1 n (x i µ x ) (x i µ x ) T i=1 1 x <- matrix ( rnorm (30), nrow =10) 2 3 # least recommended 4 cm <- colmeans ( x) 5 crossprod ( sweep (x, MARGIN =2, STATS =cm)) 6 7 # less recommended 8 crossprod ( scale (x, center =TRUE, scale = FALSE )) 9 10 # recommended 11 cov (x) pbdr Core Team Introduction to pbdr 17 / 99
26 R Syntax for Data Science: Not A Matlab Clone! Advanced Statistics (1 of 2): Principal Components PCA = centering + scaling + rotation (via SVD) 1 R> x <- matrix ( rnorm (30), nrow =10) 2 3 R> prcomp (x, retx =TRUE, scale = TRUE ) 4 Standard deviations : 5 [1] Rotation : 8 PC1 PC2 PC3 9 [1,] [2,] [3,] pbdr Core Team Introduction to pbdr 18 / 99
27 R Syntax for Data Science: Not A Matlab Clone! Advanced Statistics (2 of 2): k-means Clustering R> x <- rbind ( matrix ( rnorm (5*2, mean =0), ncol =2), 2 matrix ( rnorm (3*2, mean =10), ncol =2) ) pbdr Core Team Introduction to pbdr 19 / 99
28 R Syntax for Data Science: Not A Matlab Clone! Advanced Statistics (2 of 2): k-means Clustering 1 R> kmeans (x, centers =2) 2 K- means clustering with 2 clusters of sizes 5, Cluster means : 5 [,1] [,2] Clustering vector : 10 [1] Within cluster sum of squares by cluster : 13 [1] ( between _ SS / total _ SS = 93.9 %) Available components : [1] " cluster " " centers " " totss " " withinss " " tot. withinss " 19 [6] " betweenss " " size " pbdr Core Team Introduction to pbdr 20 / 99
29 Contents 2 pbdr The pbdr Project pbdr Paradigms pbdr Core Team Introduction to pbdr
30 The pbdr Project Programming with Big Data in R (pbdr) Striving for Productivity, Portability, Performance Bridging high-performance C with high-productivity of R Scalable, big data analytics. Free a R packages. Distributed data details implicitly managed. Methods have syntax identical to R. Powered by state of the art numerical libraries (MPI, ScaLAPACK,... ) a MPL, BSD, and GPL licensed pbdr Core Team Introduction to pbdr 21 / 99
31 The pbdr Project pbdr Packages pbdr Core Team Introduction to pbdr 22 / 99
32 The pbdr Project pbdr on HPC Resources pbdr is currently installed and maintained on: Nautilus, UTK Kraken, UTK Darter, UTK Newton, UTK Lens, ORNL tara, UMBC If you are interested in maintaining pbdr, contact us at pbdr Core Team Introduction to pbdr 23 / 99
33 The pbdr Project Example Syntax 1 x <- x[-1, 2:5] 2 x <- log ( abs (x) + 1) 3 xtx <- t(x) %*% x 4 ans <- svd ( solve ( xtx )) Look familiar? The above runs on 1 core with R or 10,000 cores with pbdr pbdr Core Team Introduction to pbdr 24 / 99
34 The pbdr Project Least Squares Benchmark 125 Fitting y~x With Fixed Local Size of ~43.4 MiB Predictors Run Time (Seconds) Cores pbdr Core Team Introduction to pbdr 25 / 99
35 pbdr Paradigms pbdr Paradigms Programs that use pbdr utilize: Batch execution Single Program/Multiple Data (SPMD) style And generally utilize: Data Parallelism pbdr Core Team Introduction to pbdr 26 / 99
36 pbdr Paradigms Batch Execution Non-interactive Use 1 Rscript my_ script. r or 1 R CMD BATCH my_ script. r In parallel: 1 mpirun - np 2 Rscript my_ par _ script. r pbdr Core Team Introduction to pbdr 27 / 99
37 pbdr Paradigms Single Program/Multiple Data (SPMD) Difficult to describe, easy to do... Only one program is written, executed in batch on all processors. Different processors are autonomous; there is no manager. The dominant programming model for large machines. pbdr Core Team Introduction to pbdr 28 / 99
38 Contents 3 Introduction to pbdmpi Managing a Communicator Reduce, Gather, Broadcast, and Barrier Other pbdmpi Tools pbdr Core Team Introduction to pbdr
39 Managing a Communicator Message Passing Interface (MPI) MPI : Standard for managing communications (data and instructions) between different nodes/computers. Implementations: OpenMPI, MPICH2, Cray MPT,... Enables parallelism (via communication) on distributed machines. Communicator: manages communications between processors. pbdr Core Team Introduction to pbdr 29 / 99
40 Managing a Communicator MPI Operations (1 of 2) Managing a Communicator: Create and destroy communicators. init() initialize communicator finalize() shut down communicator(s) Rank query: determine the processor s position in the communicator. comm.rank() who am I? comm.size() how many of us are there? Printing: Printing output from various ranks. comm.print(x) comm.cat(x) WARNING: only use these functions on results, never on yet-to-be-computed things. pbdr Core Team Introduction to pbdr 30 / 99
41 Managing a Communicator Quick Example 1 Rank Query: 1 rank.r 1 library ( pbdmpi, quiet = TRUE ) 2 init () 3 4 my. rank <- comm. rank () 5 comm. print (my.rank, all. rank = TRUE ) 6 7 finalize () Execute this script via: 1 mpirun - np 2 Rscript 1_ rank. r Sample Output: 1 COMM. RANK = 0 2 [1] 0 3 COMM. RANK = 1 4 [1] 1 pbdr Core Team Introduction to pbdr 31 / 99
42 Managing a Communicator Quick Example 2 1 library ( pbdmpi, quiet = TRUE ) 2 init () 3 4 comm. print (" Hello, world ") 5 Hello World: 2 hello.r 6 comm. print (" Hello again ", all. rank =TRUE, quiet = TRUE ) 7 8 finalize () Execute this script via: 1 mpirun - np 2 Rscript 2_ hello. r Sample Output: 1 COMM. RANK = 0 2 [1] " Hello, world " 3 [1] " Hello again " 4 [1] " Hello again " pbdr Core Team Introduction to pbdr 32 / 99
43 Reduce, Gather, Broadcast, and Barrier MPI Operations 1 Reduce 2 Gather 3 Broadcast 4 Barrier pbdr Core Team Introduction to pbdr 33 / 99
44 Reduce, Gather, Broadcast, and Barrier Reductions Combine results into single result pbdr Core Team Introduction to pbdr 34 / 99
45 Reduce, Gather, Broadcast, and Barrier Gather Many-to-one pbdr Core Team Introduction to pbdr 35 / 99
46 Reduce, Gather, Broadcast, and Barrier Broadcast One-to-many pbdr Core Team Introduction to pbdr 36 / 99
47 Reduce, Gather, Broadcast, and Barrier Barrier Synchronization pbdr Core Team Introduction to pbdr 37 / 99
48 Reduce, Gather, Broadcast, and Barrier MPI Operations (2 of 2) Reduction: each processor has a number x; add all of them up, find the largest/smallest,.... reduce(x, op= sum ) reduce to one allreduce(x, op= sum ) reduce to all Gather: each processor has a number; create a new object on some processor containing all of those numbers. gather(x) gather to one allgather(x) gather to all Broadcast: one processor has a number x that every other processor should also have. bcast(x) Barrier: computation wall ; no processor can proceed until all processors can proceed. barrier() pbdr Core Team Introduction to pbdr 38 / 99
49 Other pbdmpi Tools MPI Package Controls The.SPMD.CT object allows for setting different package options with pbdmpi. See the entry SPMD Control of the pbdmpi manual for information about the.spmd.ct object: pbdr Core Team Introduction to pbdr 39 / 99
50 Other pbdmpi Tools Random Seeds pbdmpi offers a simple interface for managing random seeds: comm.set.seed(diff=true) Independent streams via the rlecuyer package. comm.set.seed(seed=1234, diff=false) All processors use the same seed seed=1234 comm.set.seed(diff=false) All processors use the same seed, determined by processor 0 (using the system clock and PID of processor 0). pbdr Core Team Introduction to pbdr 40 / 99
51 Other pbdmpi Tools Other Helper Tools pbdmpi Also contains useful tools for Manager/Worker and task parallelism codes: Task Subsetting: Distributing a list of jobs/tasks get.jid(n) *ply: Functions in the *ply family. pbdapply(x, MARGIN, FUN,...) analogue of apply() pbdlapply(x, FUN,...) analogue of lapply() pbdsapply(x, FUN,...) analogue of sapply() pbdr Core Team Introduction to pbdr 41 / 99
52 Other pbdmpi Tools Quick Comments for Using pbdmpi 1 Start by loading the package: 1 library ( pbdmpi, quiet = TRUE ) 2 Always initialize before starting and finalize when finished: 1 init () 2 3 # finalize () pbdr Core Team Introduction to pbdr 42 / 99
53 Other pbdmpi Tools Basic MPI Exercises 1 Experiment with Quick Examples 1 through 6, running them on 2, 4, and 8 processors. pbdr Core Team Introduction to pbdr 43 / 99
54 Contents 4 The Generalized Block Distribution The GBD Data Structure GBD: Example 1 GBD: Example 2 pbdr Core Team Introduction to pbdr
55 The GBD Data Structure Distributing Data Problem: How to distribute the data x = x 1,1 x 1,2 x 1,3 x 2,1 x 2,2 x 2,3 x 3,1 x 3,2 x 3,3 x 4,1 x 4,2 x 4,3 x 5,1 x 5,2 x 5,3 x 6,1 x 6,2 x 6,3 x 7,1 x 7,2 x 7,3 x 8,1 x 8,2 x 8,3 x 9,1 x 9,2 x 9,3 x 10,1 x 10,2 x 10,3 10 3? pbdr Core Team Introduction to pbdr 44 / 99
56 The GBD Data Structure Distributing a Matrix Across 4 Processors: Block Distribution x = Data x 1,1 x 1,2 x 1,3 x 2,1 x 2,2 x 2,3 x 3,1 x 3,2 x 3,3 x 4,1 x 4,2 x 4,3 x 5,1 x 5,2 x 5,3 x 6,1 x 6,2 x 6,3 x 7,1 x 7,2 x 7,3 x 8,1 x 8,2 x 8,3 x 9,1 x 9,2 x 9,3 x 10,1 x 10,2 x 10, Processors pbdr Core Team Introduction to pbdr 45 / 99
57 The GBD Data Structure Distributing a Matrix Across 4 Processors: Local Load Balance x = Data x 1,1 x 1,2 x 1,3 x 2,1 x 2,2 x 2,3 x 3,1 x 3,2 x 3,3 x 4,1 x 4,2 x 4,3 x 5,1 x 5,2 x 5,3 x 6,1 x 6,2 x 6,3 x 7,1 x 7,2 x 7,3 x 8,1 x 8,2 x 8,3 x 9,1 x 9,2 x 9,3 x 10,1 x 10,2 x 10, Processors pbdr Core Team Introduction to pbdr 46 / 99
58 The GBD Data Structure The GBD Data Structure Throughout the examples, we will make use of the Generalized Block Distribution, or GBD distributed matrix structure. 1 GBD is distributed. No processor owns all the data. 2 GBD is non-overlapping. Rows uniquely assigned to processors. 3 GBD is row-contiguous. If a processor owns one element of a row, it owns the entire row. 4 GBD is globally row-major, locally column-major. 5 GBD is often locally balanced, where each processor owns (almost) the same amount of data. But this is not required. x 1,1 x 1,2 x 1,3 x 2,1 x 2,2 x 2,3 x 3,1 x 3,2 x 3,3 x 4,1 x 4,2 x 4,3 x 5,1 x 5,2 x 5,3 x 6,1 x 6,2 x 6,3 x 7,1 x 7,2 x 7,3 x 8,1 x 8,2 x 8,3 x 9,1 x 9,2 x 9,3 x 10,1 x 10,2 x 10,3 6 The last row of the local storage of a processor is adjacent (by global row) to the first row of the local storage of next processor (by communicator number) that owns data. 7 GBD is (relatively) easy to understand, but can lead to bottlenecks if you have many more columns than rows. pbdr Core Team Introduction to pbdr 47 / 99
59 GBD: Example 1 Understanding GBD: Global Matrix x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x Processors = pbdr Core Team Introduction to pbdr 48 / 99
60 GBD: Example 1 Understanding GBD: Load Balanced GBD x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x Processors = pbdr Core Team Introduction to pbdr 49 / 99
61 GBD: Example 1 Understanding GBD: Local View [ x11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 [ x31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 [ x51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 ] ] ] x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 [ ] x71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x [ x81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 ] 1 9 [ x91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x 99 ] 1 9 Processors = pbdr Core Team Introduction to pbdr 50 / 99
62 GBD: Example 2 Understanding GBD: Non-Balanced GBD x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x Processors = pbdr Core Team Introduction to pbdr 51 / 99
63 GBD: Example 2 Understanding GBD: Local View [ ] x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 [ x51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 [ ] x71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 ] [ ] [ x81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 ] 0 9 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x Processors = pbdr Core Team Introduction to pbdr 52 / 99
64 GBD: Example 2 Quick Comment for GBD Local pieces of GBD distributed objects will be given the suffix.gbd to visually help distinguish them from global objects. This suffix carries no semantic meaning. pbdr Core Team Introduction to pbdr 53 / 99
65 Contents 5 Basic Statistics Examples pbdmpi Example: Monte Carlo Simulation pbdmpi Example: Sample Covariance pbdmpi Example: Linear Regression pbdr Core Team Introduction to pbdr
66 pbdmpi Example: Monte Carlo Simulation Example 1: Monte Carlo Simulation Sample N uniform observations (x i, y i ) in the unit square [0, 1] [0, 1]. Then π 4 ( # Inside Circle # Total ) = 4 ( # Blue # Blue + # Red ) pbdr Core Team Introduction to pbdr 54 / 99
67 pbdmpi Example: Monte Carlo Simulation Example 1: Monte Carlo Simulation GBD Algorithm 1 Let n be big-ish; we ll take n = 50, Generate an n 2 matrix x of standard uniform observations. 3 Count the number of rows satisfying x 2 + y Ask everyone else what their answer is; sum it all up. 5 Take this new answer, multiply by 4 and divide by n 6 If my rank is 0, print the result. pbdr Core Team Introduction to pbdr 55 / 99
68 pbdmpi Example: Monte Carlo Simulation Example 1: Monte Carlo Simulation Code Serial Code 1 N < X <- matrix ( runif (N * 2), ncol =2) 3 r <- sum ( rowsums (X^2) <= 1) 4 PI <- 4*r/N 5 print (PI) Parallel Code 1 library ( pbdmpi, quiet = TRUE ) 2 init () 3 comm. set. seed ( diff = TRUE ) 4 5 N. gbd < / comm. size () 6 X. gbd <- matrix ( runif (N. gbd * 2), ncol = 2) 7 r. gbd <- sum ( rowsums (X. gbd ^2) <= 1) 8 r <- allreduce (r. gbd ) 9 PI <- 4*r/(N. gbd * comm. size ()) 10 comm. print ( PI) finalize () pbdr Core Team Introduction to pbdr 56 / 99
69 pbdmpi Example: Monte Carlo Simulation Note For the remainder, we will exclude loading, init, and finalize calls. pbdr Core Team Introduction to pbdr 57 / 99
70 pbdmpi Example: Sample Covariance Example 2: Sample Covariance cov(x n p ) = 1 n 1 n (x i µ x ) (x i µ x ) T i=1 pbdr Core Team Introduction to pbdr 58 / 99
71 pbdmpi Example: Sample Covariance Example 2: Sample Covariance GBD Algorithm 1 Determine the total number of rows N. 2 Compute the vector of column means of the full matrix. 3 Subtract each column s mean from that column s entries in each local matrix. 4 Compute the crossproduct locally and reduce. 5 Divide by N 1. pbdr Core Team Introduction to pbdr 59 / 99
72 pbdmpi Example: Sample Covariance Example 2: Sample Covariance Code 1 N <- nrow ( X) 2 mu <- colsums ( X) / N 3 Serial Code 4 X <- sweep (X, STATS =mu, MARGIN =2) 5 Cov. X <- crossprod ( X) / (N -1) 6 7 print ( Cov.X) Parallel Code 1 N <- allreduce ( nrow (X. gbd ), op=" sum ") 2 mu <- allreduce ( colsums (X. gbd ) / N, op=" sum ") 3 4 X. gbd <- sweep (X.gbd, STATS =mu, MARGIN =2) 5 Cov.X <- allreduce ( crossprod (X. gbd ), op=" sum ") / (N -1) 6 7 comm. print ( Cov.X) pbdr Core Team Introduction to pbdr 60 / 99
73 pbdmpi Example: Linear Regression Example 3: Linear Regression Find β such that y = Xβ + ɛ When X is full rank, ˆβ = (X T X) 1 X T y pbdr Core Team Introduction to pbdr 61 / 99
74 pbdmpi Example: Linear Regression Example 3: Linear Regression GBD Algorithm 1 Locally, compute tx = x T 2 Locally, compute A = tx x. Query every other processor for this result and sum up all the results. 3 Locally, compute B = tx y. Query every other processor for this result and sum up all the results. 4 Locally, compute A 1 B pbdr Core Team Introduction to pbdr 62 / 99
75 pbdmpi Example: Linear Regression Example 3: Linear Regression Code 1 tx <- t( X) 2 A <- tx %*% X 3 B <- tx %*% y 4 5 ols <- solve (A) %*% B Serial Code Parallel Code 1 tx.gbd <- t(x. gbd ) 2 A <- allreduce (tx.gbd %*% X.gbd, op = " sum ") 3 B <- allreduce (tx.gbd %*% y.gbd, op = " sum ") 4 5 ols <- solve (A) %*% B pbdr Core Team Introduction to pbdr 63 / 99
76 pbdmpi Example: Linear Regression MPI Exercises 1 Experiment with Statistics Examples 1 through 3, running them on 2, 4, and 8 processors. pbdr Core Team Introduction to pbdr 64 / 99
77 pbdmpi Example: Linear Regression Advanced MPI Exercises I 1 Write a script that will have each processor randomly take a sample of size 1 of TRUE and FALSE. Have each processor print its result. 2 Modify the script in Exercise 1 above to determine if any processors sampled TRUE. Do the same to determine if all processors sampled TRUE. In each case, print the result. Compare to the functions comm.all() and comm.any(). 3 Generate 50, 000, 000 (total) random normal values in parallel on 2, 4, and 8 processors. Time each run. pbdr Core Team Introduction to pbdr 65 / 99
78 pbdmpi Example: Linear Regression Advanced MPI Exercises II 4 Distribute the matrix x <- matrix(1:24, nrow=12) in GBD format across 4 processors and call it x.spmd. 1 Add x.spmd to itself. 2 Compute the mean of x.spmd. 3 Compute the column means of x.spmd. pbdr Core Team Introduction to pbdr 66 / 99
79 Contents 6 Introduction to pbddmat and the DMAT Structure Introduction to Distributed Matrices DMAT Distributions pbddmat pbdr Core Team Introduction to pbdr
80 Introduction to Distributed Matrices Distributed Matrices Most problems in data science are matrix algebra problems, so: Distributed matrices = Handle Bigger data pbdr Core Team Introduction to pbdr 67 / 99
81 Introduction to Distributed Matrices Distributed Matrices High level OOP allows native serial R syntax: 1 x <- x[-1, 2:5] 2 x <- log ( abs (x) + 1) 3 xtx <- t(x) %*% x 4 ans <- svd ( solve ( xtx )) However... pbdr Core Team Introduction to pbdr 68 / 99
82 Introduction to Distributed Matrices Distributed Matrices DMAT: Distributed MATrix data structure. No single processor should hold all of the data. Block-cyclic matrix distributed across a 2-dimensional grid of processors. Very robust, but confusing data structure. pbdr Core Team Introduction to pbdr 69 / 99
83 Introduction to Distributed Matrices Distributed Matrices (a) Block (b) Cyclic (c) Block-Cyclic Figure: Matrix Distribution Schemes pbdr Core Team Introduction to pbdr 70 / 99
84 Introduction to Distributed Matrices Distributed Matrices (a) 2d Block (b) 2d Cyclic (c) 2d Block-Cyclic Figure: Matrix Distribution Schemes Onto a 2-Dimensional Grid pbdr Core Team Introduction to pbdr 71 / 99
85 Introduction to Distributed Matrices Processor Grid Shapes T [ (a) 1 6 (b) 2 3 ] (c) 3 2 Table: Processor Grid Shapes with 6 Processors (d) pbdr Core Team Introduction to pbdr 72 / 99
86 Introduction to Distributed Matrices Distributed Matrices The data structure is a special R class (in the OOP sense) called ddmatrix. It is the under the rug storage for a block-cyclic matrix distributed onto a 2-dimensional processor grid. Data dim ddmatrix = ldim bldim CTXT S4 local submatrix, an R matrix S4 dimension of the global matrix, a numeric pair S4 dimension of the local submatrix, a numeric pair S4 ScaLAPACK blocking factor, a numeric pair S4 BLACS context, an numeric singleton with prototype Data = matrix(0.0) dim = c(1,1) new("ddmatrix") = ldim = c(1,1) bldim = c(1,1) CTXT = pbdr Core Team Introduction to pbdr 73 / 99
87 Introduction to Distributed Matrices Distributed Matrices: The Data Structure Example: an 9 9 matrix is distributed with a block-cycling factor of 2 2 on a 2 2 processor grid: Data = matrix(...) dim = c(9, 9) = ldim = c(...) bldim = c(2, 2) CTXT = 0 See pbdr Core Team Introduction to pbdr 74 / 99
88 DMAT Distributions Understanding Dmat: Global Matrix x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x pbdr Core Team Introduction to pbdr 75 / 99
89 DMAT Distributions DMAT: 1-dimensional Row Block x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x 99 Processor grid = = (0,0) (0,1) (1,0) (1,1) pbdr Core Team Introduction to pbdr 76 /
90 DMAT Distributions DMAT: 2-dimensional Row Block x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x Processor grid = = (0,0) (0,1) (1,0) (1,1) pbdr Core Team Introduction to pbdr 77 / 99
91 DMAT Distributions DMAT: 1-dimensional Row Cyclic x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x 99 Processor grid = = (0,0) (0,1) (1,0) (1,1) pbdr Core Team Introduction to pbdr 78 /
92 DMAT Distributions DMAT: 2-dimensional Row Cyclic x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x Processor grid = = (0,0) (0,1) (1,0) (1,1) pbdr Core Team Introduction to pbdr 79 / 99
93 DMAT Distributions DMAT: 2-dimensional Block-Cyclic x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x Processor grid = = (0,0) (0,1) (1,0) (1,1) pbdr Core Team Introduction to pbdr 80 / 99
94 pbddmat The DMAT Data Structure The more complicated the processor grid, the more complicated the distribution. pbdr Core Team Introduction to pbdr 81 / 99
95 pbddmat DMAT: 2-dimensional Block-Cyclic with 6 Processors x = x 11 x 12 x 13 x 14 x 15 x 16 x 17 x 18 x 19 x 21 x 22 x 23 x 24 x 25 x 26 x 27 x 28 x 29 x 31 x 32 x 33 x 34 x 35 x 36 x 37 x 38 x 39 x 41 x 42 x 43 x 44 x 45 x 46 x 47 x 48 x 49 x 51 x 52 x 53 x 54 x 55 x 56 x 57 x 58 x 59 x 61 x 62 x 63 x 64 x 65 x 66 x 67 x 68 x 69 x 71 x 72 x 73 x 74 x 75 x 76 x 77 x 78 x 79 x 81 x 82 x 83 x 84 x 85 x 86 x 87 x 88 x 89 x 91 x 92 x 93 x 94 x 95 x 96 x 97 x 98 x Processor grid = = (0,0) (0,1) (0,2) (1,0) (1,1) (1,2) pbdr Core Team Introduction to pbdr 82 / 99
96 pbddmat Understanding DMAT: Local View x 11 x 12 x 17 x 18 x 21 x 22 x 27 x 28 x 51 x 52 x 57 x 58 x 61 x 62 x 67 x 68 x 91 x 92 x 97 x 98 x 31 x 32 x 37 x 38 x 41 x 42 x 47 x 48 x 71 x 72 x 77 x 78 x 81 x 82 x 87 x x 13 x 14 x 19 x 23 x 24 x 29 x 53 x 54 x 59 x 63 x 64 x 69 x 93 x 94 x 99 x 33 x 34 x 39 x 43 x 44 x 49 x 73 x 74 x 79 x 83 x 84 x x 15 x 16 x 25 x 26 x 55 x 56 x 65 x 66 x 95 x 96 x 35 x 36 x 45 x 46 x 75 x 76 x 85 x Processor grid = = (0,0) (0,1) (0,2) (1,0) (1,1) (1,2) pbdr Core Team Introduction to pbdr 83 / 99
97 pbddmat The DMAT Data Structure 1 DMAT is distributed. No one processor owns all of the matrix. 2 DMAT is non-overlapping. Any piece owned by one processor is owned by no other processors. 3 DMAT can be row-contiguous or not, depending on the processor grid and blocking factor used. 4 DMAT is locally column-major and globally, it depends... x 11 x 12 x 13 x 14 x 15 x 21 x 22 x 23 x 24 x 25 x 31 x 32 x 33 x 34 x 35 x 41 x 42 x 43 x 44 x 45 x 51 x 52 x 53 x 54 x 55 x 61 x 62 x 63 x 64 x 65 x 71 x 72 x 73 x 74 x 75 x 81 x 82 x 83 x 84 x 85 x 91 x 92 x 93 x 94 x 95 6 GBD is a generalization of the one-dimensional block DMAT distribution. Otherwise there is no relation. 7 DMAT is confusing, but very robust. pbdr Core Team Introduction to pbdr 84 / 99
98 pbddmat Pros and Cons of This Data Structure Pros Fast for distributed matrix computations Cons Literally everything else This is why we hide most of the distributed details. The details are there if you want them (you don t want them). pbdr Core Team Introduction to pbdr 85 / 99
99 pbddmat Distributed Matrix Methods pbddmat has over 100 methods with identical syntax to R: `[`, rbind(), cbind(),... lm.fit(), prcomp(), cov(),... `%*%`, solve(), svd(), norm(),... median(), mean(), rowsums(),... 1 cov (x) 1 cov (x) Serial Code Parallel Code pbdr Core Team Introduction to pbdr 86 / 99
100 pbddmat Comparing pbdmpi and pbddmat pbdmpi: MPI + sugar. GBD not the only structure pbdmpi can handle (just a useful convention). pbddmat: More of a software package. DMAT structure must be used for pbddmat. If the data is not 2d block-cyclic compatible, DMAT will definitely give the wrong answer. pbdr Core Team Introduction to pbdr 87 / 99
101 pbddmat Quick Comments for Using pbddmat 1 Start by loading the package: 1 library ( pbddmat, quiet = TRUE ) 2 Always initialize before starting and finalize when finished: 1 init. grid () 2 3 # finalize () 3 Distributed DMAT objects will be given the suffix.dmat to visually help distinguish them from global objects. This suffix carries no semantic meaning. pbdr Core Team Introduction to pbdr 88 / 99
102 Contents 7 Examples Using pbddmat Statistics Examples with pbddmat RandSVD pbdr Core Team Introduction to pbdr
103 Statistics Examples with pbddmat Sample Covariance 1 Cov.X <- cov (X) 2 print ( Cov.X) 1 Cov.X <- cov (X) 2 print ( Cov.X) Serial Code Parallel Code pbdr Core Team Introduction to pbdr 89 / 99
104 Statistics Examples with pbddmat Linear Regression 1 tx <- t(x) 2 A <- tx %*% X 3 B <- tx %*% y 4 5 ols <- solve (A) %*% B 6 7 # or 8 ols <- lm. fit (X, y) 1 tx <- t(x) 2 A <- tx %*% X 3 B <- tx %*% y 4 5 ols <- solve (A) %*% B 6 7 # or 8 ols <- lm. fit (X, y) Serial Code Parallel Code pbdr Core Team Introduction to pbdr 90 / 99
105 Statistics Examples with pbddmat Example 5: PCA PCA: pca.r 1 l i b r a r y (pbddmat, q u i e t=t) 2 i n i t. g r i d ( ) 3 4 n < 1 e4 5 p < comm. s e t. s e e d ( d i f f =T) 8 x. dmat < ddmatrix ( rnorm, nrow=n, ncol=p, mean=100, sd=25) 9 10 pca < prcomp ( x=x. dmat, r e t x=true, s c a l e=true) 11 prop var < cumsum ( pca$ sdev ) /sum ( pca$ sdev ) 12 i < max ( min ( which ( prop v a r > 0. 9 ) ) 1, 1) y. dmat < pca $ x [, 1 : i ] comm. c a t ( \ n C o l s :, i, \n, q u i e t=t) 17 comm. c a t ( %C o l s :, i /dim ( x. dmat ) [ 2 ], \n\n, q u i e t=t) f i n a l i z e ( ) Execute this script via: 1 mpirun np 2 R s c r i p t 5 pca. r 1 Cols : %Cols : Sample Output: pbdr Core Team Introduction to pbdr 91 / 99
106 Statistics Examples with pbddmat Distributed Matrices pbddemo contains many other examples of reading and managing GBD and DMAT data pbdr Core Team Introduction to pbdr 92 / 99
107 RandSVD Randomized SVD 1 PROBABILISTIC ALGORITHMS FOR MATRIX APPROXIMATION 227 Serial R Prototype for Randomized SVD Given an m n matrix A, a target number k of singular vectors, and an exponent q (say, q =1or q =2), this procedure computes an approximate 1 randsvd < f u n c t i o n (A, k, q=3) rank-2k factorization UΣV,whereU and V are orthonormal, and Σ is 2 { nonnegative and diagonal. 3 ## Stage A Stage A: 244 N. HALKO, P. G. MARTINSSON, AND J. A. TROPP 1 Generate an n 2k Gaussian test matrix Ω. 4 Omega < matrix(rnorm(n*2*k), 2 Form Y =(AA ) q AΩ by multiplying alternately with A and A. 5 nrow=n, ncol=2*k) 3 Construct Algorithm a matrix Q4.3: whose Randomized columns formpower an orthonormal Iterationbasis for Given thean range m of ny matrix. A and integers l and q, this algorithm computes an 6 Y < A % % Omega Stage m l orthonormal B: matrix Q whose range approximates the range of A. 7 Q < qr.q( qr (Y) ) 4 1 Form Draw B an= n Q la. Gaussian random matrix Ω. 8 At < t (A) 2 5 Compute Form the an m SVD l matrix of the Y small =(AA matrix: ) q AΩ B via = ŨΣV alternating. application 9 f o r ( i i n 1 : q ) of A and 6 Set U = QŨ. A. 3 Construct an m l matrix Q whose columns form an orthonormal 10 { Note: The computation of Y in step 2 is vulnerable to round-off errors. basis for the range of Y, e.g., via the QR factorization Y = QR. 11 Y < At % % Q When high accuracy is required, we must incorporate an orthonormalization Note: This procedure is vulnerable to round-off step between each application of A and A errors; see Remark 4.3. The ; see Algorithm Q < qr.q( qr (Y) ) recommended implementation appears as Algorithm Y < A % % Q 14 Q < qr.q( qr (Y) ) The theoryalgorithm developed in4.4: thisrandomized paper provides Subspace much more Iteration detailed information about Given the performance m n matrix of thea proto-algorithm. and integers l and q, this algorithm computes an 15 } m When l orthonormal the singular matrix values Q whose of A decay range slightly, approximates the error the range A of QQA. A does 16 1 not Draw depend an n onl the standard dimensions Gaussian of the matrix matrix Ω. (sections ). 17 ## Stage B 2 We Formcan reduce the size of the bracket in the error bound (1.8) by combining Y0 = AΩ and compute its QR factorization Y0 = Q0R0. the proto-algorithm with a power iteration (section 10.4). For an example, 18 B < t (Q) % % A 3 for j =1, 2,..., q 4 see section Form 1.6 below. For the structured Qj 1 and compute its QR factorization Ỹj = Qj Rj. 19 U < La. svd (B) $u random matrices we mentioned in section 1.4.1, related 5 Form Yj = A error bounds are in and compute its QR factorization Yj = QjRj. 20 U < Q % % U force (section 11). 6 end 21 U[, 1 : k ] We can obtain inexpensive a posteriori error estimates to verify the quality 7 Q = Qq. of the approximation (section 4.3). 22 } 1.6. Example: Randomized SVD. We conclude this introduction with a short 1discussion Algorithm of how4.3 these targets ideas the allow fixed-rank us to perform problem. an approximate To addresssvd the fixed-precision of a large data problem, matrix, Halko which N, Martinsson we can is a incorporate compelling P-G the application and error estimators of Tropp randomized J A described matrix 2011 in section approximation Finding structure 4.3 to obtain [113]. with randomness: probabilistic algorithms for an adaptive The two-stage schemerandomized analogous with method Algorithm offers4.2. a natural In situations approach where to itsvd is critical computations. near-optimal Unfortunately, approximation the simplesterrors, version oneof can this increase schemethe is oversampling inadequate inbeyond many constructing approximate matrix decompositions SIAM Rev. 53 to achieve our applications standard because recommendation the singular l spectrum = k +5 all of the the input way to matrix l =2k may without decay slowly. changing To pbdr Core Team Introduction to pbdr 93 / 99 address this difficulty, we incorporate q steps of a power iteration, where q =1or
108 RandSVD Randomized SVD Serial R 1 randsvd < function (A, k, q=3) 2 { 3 ## Stage A 4 Omega < matrix(rnorm(n*2*k), 5 nrow=n, ncol=2*k) 6 Y < A % % Omega 7 Q < qr.q( qr (Y) ) 8 At < t (A) 9 f o r ( i i n 1 : q ) 10 { 11 Y < At % % Q 12 Q < qr.q( qr (Y) ) 13 Y < A % % Q 14 Q < qr.q( qr (Y) ) 15 } ## Stage B 18 B < t (Q) % % A 19 U < La. svd (B) $u 20 U < Q % % U 21 U[, 1 : k ] 22 } Parallel pbdr 1 randsvd < function (A, k, q=3) 2 { 3 ## Stage A 4 Omega < ddmatrix( rnorm, 5 nrow=n, ncol=2*k) 6 Y < A % % Omega 7 Q < qr.q( qr (Y) ) 8 At < t (A) 9 f o r ( i i n 1 : q ) 10 { 11 Y < At % % Q 12 Q < qr.q( qr (Y) ) 13 Y < A % % Q 14 Q < qr.q( qr (Y) ) 15 } ## Stage B 18 B < t (Q) % % A 19 U < La. svd (B) $u 20 U < Q % % U 21 U[, 1 : k ] 22 } pbdr Core Team Introduction to pbdr 94 / 99
109 RandSVD Randomized SVD 30 Singular Vectors from a 100,000 by 1,000 Matrix Algorithm full randomized 30 Singular Vectors from a 100,000 by 1,000 Matrix Speedup of Randomized vs. Full SVD Speedup Speedup Cores Cores pbdr Core Team Introduction to pbdr 95 / 99
110 RandSVD DMAT Exercises 1 Experiment with DMAT Examples 1 through 5, running them on 2 and 4 processors. pbdr Core Team Introduction to pbdr 96 / 99
111 RandSVD Advanced DMAT Exercises I 1 Subsetting, selection, and filtering are basic matrix operations featured in R. The following may look silly, but it is useful for data processing. Let x.dmat <- ddmatrix(1:30, 10, 3). Do the following: y.dmat <- x.dmat[c(1, 5, 4, 3), ] y.dmat <- x.dmat[c(10:3, 5, 5), ] y.dmat <- x.dmat[1:5, 3:1] y.dmat <- x.dmat[x.dmat[, 2] > 13, ] y.dmat <- x.dmat[x.dmat[, 2] > x.dmat[, 3], ] y.dmat <- x.dmat[, x.dmat[2,] > x.dmat[3, ]] y.dmat <- x.dmat[c(1, 3, 5), x.dmat[, 2] > x.dmat[, 3]] pbdr Core Team Introduction to pbdr 97 / 99
112 RandSVD Advanced DMAT Exercises II 2 The method crossprod() is an optimized form of the crossproduct computation t(x.dmat) %*% x.dmat. For this exercise, let x.dmat <- ddmatrix(1:30, nrow=10, ncol=3). 1 Verify that these computations really do produce the same results. 2 Time each operation. Which is faster? 3 The prcomp() method returns rotations for all components. Computationally verify by example that these rotations are orthogonal, i.e., that their crossproduct is the identity matrix. pbdr Core Team Introduction to pbdr 98 / 99
113 Contents 8 Wrapup pbdr Core Team Introduction to pbdr
114 The pbdr Project Our website: us at: Our google group: Where to begin? The pbddemo package The pbddemo Vignette: pbdr Core Team Introduction to pbdr 99 / 99
115 Thanks for coming! Questions? pbdr Core Team Introduction to pbdr
Programming with Big Data in R
Programming with Big Data in R Drew Schmidt and George Ostrouchov user! 2014 http://r-pbd.org/tutorial pbdr Core Team Programming with Big Data in R The pbdr Core Team Wei-Chen Chen 1 George Ostrouchov
Version 0.2-0. Programming with Big Data in R. Speaking Serial R with a Parallel Accent. Package Examples and Demonstrations.
Version 0.2-0 Programming with Big Data in R Speaking Serial R with a Parallel Accent Package Examples and Demonstrations pbdr Core Team Speaking Serial R with a Parallel Accent (Ver. 0.2-0) pbdr Package
APPM4720/5720: Fast algorithms for big data. Gunnar Martinsson The University of Colorado at Boulder
APPM4720/5720: Fast algorithms for big data Gunnar Martinsson The University of Colorado at Boulder Course objectives: The purpose of this course is to teach efficient algorithms for processing very large
CUDAMat: a CUDA-based matrix class for Python
Department of Computer Science 6 King s College Rd, Toronto University of Toronto M5S 3G4, Canada http://learning.cs.toronto.edu fax: +1 416 978 1455 November 25, 2009 UTML TR 2009 004 CUDAMat: a CUDA-based
Big Data, R, and HPC
Big Data, R, and HPC A Survey of Advanced Computing with R Drew Schmidt April 16, 2015 Drew Schmidt Big Data, R, and HPC About Me @wrathematics http://librestats.com https://github.com/wrathematics http://wrathematics.info
Linear Algebra Review. Vectors
Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka [email protected] http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length
Lecture 5: Singular Value Decomposition SVD (1)
EEM3L1: Numerical and Analytical Techniques Lecture 5: Singular Value Decomposition SVD (1) EE3L1, slide 1, Version 4: 25-Sep-02 Motivation for SVD (1) SVD = Singular Value Decomposition Consider the system
Introduction to parallel computing in R
Introduction to parallel computing in R Clint Leach April 10, 2014 1 Motivation When working with R, you will often encounter situations in which you need to repeat a computation, or a series of computations,
The Assessment of Benchmarks Executed on Bare-Metal and Using Para-Virtualisation
The Assessment of Benchmarks Executed on Bare-Metal and Using Para-Virtualisation Mark Baker, Garry Smith and Ahmad Hasaan SSE, University of Reading Paravirtualization A full assessment of paravirtualization
Review Jeopardy. Blue vs. Orange. Review Jeopardy
Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round $200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?
Fast Analytics on Big Data with H20
Fast Analytics on Big Data with H20 0xdata.com, h2o.ai Tomas Nykodym, Petr Maj Team About H2O and 0xdata H2O is a platform for distributed in memory predictive analytics and machine learning Pure Java,
15.062 Data Mining: Algorithms and Applications Matrix Math Review
.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop
Package biganalytics
Package biganalytics February 19, 2015 Version 1.1.1 Date 2012-09-20 Title A library of utilities for big.matrix objects of package bigmemory. Author John W. Emerson and Michael
How To Cluster
Data Clustering Dec 2nd, 2013 Kyrylo Bessonov Talk outline Introduction to clustering Types of clustering Supervised Unsupervised Similarity measures Main clustering algorithms k-means Hierarchical Main
CS 5614: (Big) Data Management Systems. B. Aditya Prakash Lecture #18: Dimensionality Reduc7on
CS 5614: (Big) Data Management Systems B. Aditya Prakash Lecture #18: Dimensionality Reduc7on Dimensionality Reduc=on Assump=on: Data lies on or near a low d- dimensional subspace Axes of this subspace
AMATH 352 Lecture 3 MATLAB Tutorial Starting MATLAB Entering Variables
AMATH 352 Lecture 3 MATLAB Tutorial MATLAB (short for MATrix LABoratory) is a very useful piece of software for numerical analysis. It provides an environment for computation and the visualization. Learning
Operation Count; Numerical Linear Algebra
10 Operation Count; Numerical Linear Algebra 10.1 Introduction Many computations are limited simply by the sheer number of required additions, multiplications, or function evaluations. If floating-point
Nimble Algorithms for Cloud Computing. Ravi Kannan, Santosh Vempala and David Woodruff
Nimble Algorithms for Cloud Computing Ravi Kannan, Santosh Vempala and David Woodruff Cloud computing Data is distributed arbitrarily on many servers Parallel algorithms: time Streaming algorithms: sublinear
LINEAR ALGEBRA. September 23, 2010
LINEAR ALGEBRA September 3, 00 Contents 0. LU-decomposition.................................... 0. Inverses and Transposes................................. 0.3 Column Spaces and NullSpaces.............................
Java Modules for Time Series Analysis
Java Modules for Time Series Analysis Agenda Clustering Non-normal distributions Multifactor modeling Implied ratings Time series prediction 1. Clustering + Cluster 1 Synthetic Clustering + Time series
18.06 Problem Set 4 Solution Due Wednesday, 11 March 2009 at 4 pm in 2-106. Total: 175 points.
806 Problem Set 4 Solution Due Wednesday, March 2009 at 4 pm in 2-06 Total: 75 points Problem : A is an m n matrix of rank r Suppose there are right-hand-sides b for which A x = b has no solution (a) What
Map-Reduce for Machine Learning on Multicore
Map-Reduce for Machine Learning on Multicore Chu, et al. Problem The world is going multicore New computers - dual core to 12+-core Shift to more concurrent programming paradigms and languages Erlang,
CS3220 Lecture Notes: QR factorization and orthogonal transformations
CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss
Orthogonal Diagonalization of Symmetric Matrices
MATH10212 Linear Algebra Brief lecture notes 57 Gram Schmidt Process enables us to find an orthogonal basis of a subspace. Let u 1,..., u k be a basis of a subspace V of R n. We begin the process of finding
3 Orthogonal Vectors and Matrices
3 Orthogonal Vectors and Matrices The linear algebra portion of this course focuses on three matrix factorizations: QR factorization, singular valued decomposition (SVD), and LU factorization The first
Introduction to the data.table package in R
Introduction to the data.table package in R Revised: September 18, 2015 (A later revision may be available on the homepage) Introduction This vignette is aimed at those who are already familiar with creating
Parallel Computing using MATLAB Distributed Compute Server ZORRO HPC
Parallel Computing using MATLAB Distributed Compute Server ZORRO HPC Goals of the session Overview of parallel MATLAB Why parallel MATLAB? Multiprocessing in MATLAB Parallel MATLAB using the Parallel Computing
Text Mining in JMP with R Andrew T. Karl, Senior Management Consultant, Adsurgo LLC Heath Rushing, Principal Consultant and Co-Founder, Adsurgo LLC
Text Mining in JMP with R Andrew T. Karl, Senior Management Consultant, Adsurgo LLC Heath Rushing, Principal Consultant and Co-Founder, Adsurgo LLC 1. Introduction A popular rule of thumb suggests that
Solution of Linear Systems
Chapter 3 Solution of Linear Systems In this chapter we study algorithms for possibly the most commonly occurring problem in scientific computing, the solution of linear systems of equations. We start
:Introducing Star-P. The Open Platform for Parallel Application Development. Yoel Jacobsen E&M Computing LTD [email protected]
:Introducing Star-P The Open Platform for Parallel Application Development Yoel Jacobsen E&M Computing LTD [email protected] The case for VHLLs Functional / applicative / very high-level languages allow
High Performance Computing with R
High Performance Computing with R Drew Schmidt April 6, 2014 http://r-pbd.org/nimbios Drew Schmidt High Performance Computing with R Contents 1 Introduction 2 Profiling and Benchmarking 3 Writing Better
Financial Econometrics MFE MATLAB Introduction. Kevin Sheppard University of Oxford
Financial Econometrics MFE MATLAB Introduction Kevin Sheppard University of Oxford October 21, 2013 2007-2013 Kevin Sheppard 2 Contents Introduction i 1 Getting Started 1 2 Basic Input and Operators 5
1 Introduction to Matrices
1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns
Parallelization Strategies for Multicore Data Analysis
Parallelization Strategies for Multicore Data Analysis Wei-Chen Chen 1 Russell Zaretzki 2 1 University of Tennessee, Dept of EEB 2 University of Tennessee, Dept. Statistics, Operations, and Management
Architectures for Big Data Analytics A database perspective
Architectures for Big Data Analytics A database perspective Fernando Velez Director of Product Management Enterprise Information Management, SAP June 2013 Outline Big Data Analytics Requirements Spectrum
December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B. KITCHENS
December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B KITCHENS The equation 1 Lines in two-dimensional space (1) 2x y = 3 describes a line in two-dimensional space The coefficients of x and y in the equation
Partial Least Squares (PLS) Regression.
Partial Least Squares (PLS) Regression. Hervé Abdi 1 The University of Texas at Dallas Introduction Pls regression is a recent technique that generalizes and combines features from principal component
Text Analytics (Text Mining)
CSE 6242 / CX 4242 Apr 3, 2014 Text Analytics (Text Mining) LSI (uses SVD), Visualization Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey
RevoScaleR Speed and Scalability
EXECUTIVE WHITE PAPER RevoScaleR Speed and Scalability By Lee Edlefsen Ph.D., Chief Scientist, Revolution Analytics Abstract RevoScaleR, the Big Data predictive analytics library included with Revolution
Mehtap Ergüven Abstract of Ph.D. Dissertation for the degree of PhD of Engineering in Informatics
INTERNATIONAL BLACK SEA UNIVERSITY COMPUTER TECHNOLOGIES AND ENGINEERING FACULTY ELABORATION OF AN ALGORITHM OF DETECTING TESTS DIMENSIONALITY Mehtap Ergüven Abstract of Ph.D. Dissertation for the degree
The Singular Value Decomposition in Symmetric (Löwdin) Orthogonalization and Data Compression
The Singular Value Decomposition in Symmetric (Löwdin) Orthogonalization and Data Compression The SVD is the most generally applicable of the orthogonal-diagonal-orthogonal type matrix decompositions Every
Matrix Multiplication
Matrix Multiplication CPS343 Parallel and High Performance Computing Spring 2016 CPS343 (Parallel and HPC) Matrix Multiplication Spring 2016 1 / 32 Outline 1 Matrix operations Importance Dense and sparse
8 Square matrices continued: Determinants
8 Square matrices continued: Determinants 8. Introduction Determinants give us important information about square matrices, and, as we ll soon see, are essential for the computation of eigenvalues. You
Scalable Data Analysis in R. Lee E. Edlefsen Chief Scientist UserR! 2011
Scalable Data Analysis in R Lee E. Edlefsen Chief Scientist UserR! 2011 1 Introduction Our ability to collect and store data has rapidly been outpacing our ability to analyze it We need scalable data analysis
Load Balancing. computing a file with grayscales. granularity considerations static work load assignment with MPI
Load Balancing 1 the Mandelbrot set computing a file with grayscales 2 Static Work Load Assignment granularity considerations static work load assignment with MPI 3 Dynamic Work Load Balancing scheduling
Parallel Computing with Mathematica UVACSE Short Course
UVACSE Short Course E Hall 1 1 University of Virginia Alliance for Computational Science and Engineering [email protected] October 8, 2014 (UVACSE) October 8, 2014 1 / 46 Outline 1 NX Client for Remote
Distributed R for Big Data
Distributed R for Big Data Indrajit Roy HP Vertica Development Team Abstract Distributed R simplifies large-scale analysis. It extends R. R is a single-threaded environment which limits its utility for
Applied Linear Algebra I Review page 1
Applied Linear Algebra Review 1 I. Determinants A. Definition of a determinant 1. Using sum a. Permutations i. Sign of a permutation ii. Cycle 2. Uniqueness of the determinant function in terms of properties
Package bigrf. February 19, 2015
Version 0.1-11 Date 2014-05-16 Package bigrf February 19, 2015 Title Big Random Forests: Classification and Regression Forests for Large Data Sets Maintainer Aloysius Lim OS_type
Parallel Options for R
Parallel Options for R Glenn K. Lockwood SDSC User Services [email protected] Motivation "I just ran an intensive R script [on the supercomputer]. It's not much faster than my own machine." Motivation "I
Package dsmodellingclient
Package dsmodellingclient Maintainer Author Version 4.1.0 License GPL-3 August 20, 2015 Title DataSHIELD client site functions for statistical modelling DataSHIELD
G563 Quantitative Paleontology. SQL databases. An introduction. Department of Geological Sciences Indiana University. (c) 2012, P.
SQL databases An introduction AMP: Apache, mysql, PHP This installations installs the Apache webserver, the PHP scripting language, and the mysql database on your computer: Apache: runs in the background
Big Data Analytics: Optimization and Randomization
Big Data Analytics: Optimization and Randomization Tianbao Yang, Qihang Lin, Rong Jin Tutorial@SIGKDD 2015 Sydney, Australia Department of Computer Science, The University of Iowa, IA, USA Department of
This exam contains 13 pages (including this cover page) and 18 questions. Check to see if any pages are missing.
Big Data Processing 2013-2014 Q2 April 7, 2014 (Resit) Lecturer: Claudia Hauff Time Limit: 180 Minutes Name: Answer the questions in the spaces provided on this exam. If you run out of room for an answer,
22S:295 Seminar in Applied Statistics High Performance Computing in Statistics
22S:295 Seminar in Applied Statistics High Performance Computing in Statistics Luke Tierney Department of Statistics & Actuarial Science University of Iowa August 30, 2007 Luke Tierney (U. of Iowa) HPC
MapReduce and Hadoop. Aaron Birkland Cornell Center for Advanced Computing. January 2012
MapReduce and Hadoop Aaron Birkland Cornell Center for Advanced Computing January 2012 Motivation Simple programming model for Big Data Distributed, parallel but hides this Established success at petabyte
Eigenvalues and Eigenvectors
Chapter 6 Eigenvalues and Eigenvectors 6. Introduction to Eigenvalues Linear equations Ax D b come from steady state problems. Eigenvalues have their greatest importance in dynamic problems. The solution
Spark ΕΡΓΑΣΤΗΡΙΟ 10. Prepared by George Nikolaides 4/19/2015 1
Spark ΕΡΓΑΣΤΗΡΙΟ 10 Prepared by George Nikolaides 4/19/2015 1 Introduction to Apache Spark Another cluster computing framework Developed in the AMPLab at UC Berkeley Started in 2009 Open-sourced in 2010
Capturing Database Transformations for Big Data Analytics
Capturing Database Transformations for Big Data Analytics David Sergio Matusevich University of Houston Organization Introduction Classification Program Case Study Conclusions Extending ER Models to Capture
Cluster Analysis using R
Cluster analysis or clustering is the task of assigning a set of objects into groups (called clusters) so that the objects in the same cluster are more similar (in some sense or another) to each other
Tensor Methods for Machine Learning, Computer Vision, and Computer Graphics
Tensor Methods for Machine Learning, Computer Vision, and Computer Graphics Part I: Factorizations and Statistical Modeling/Inference Amnon Shashua School of Computer Science & Eng. The Hebrew University
Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh
Modern Optimization Methods for Big Data Problems MATH11146 The University of Edinburgh Peter Richtárik Week 3 Randomized Coordinate Descent With Arbitrary Sampling January 27, 2016 1 / 30 The Problem
Randomized Robust Linear Regression for big data applications
Randomized Robust Linear Regression for big data applications Yannis Kopsinis 1 Dept. of Informatics & Telecommunications, UoA Thursday, Apr 16, 2015 In collaboration with S. Chouvardas, Harris Georgiou,
Beginner s Matlab Tutorial
Christopher Lum [email protected] Introduction Beginner s Matlab Tutorial This document is designed to act as a tutorial for an individual who has had no prior experience with Matlab. For any questions
Machine Learning Big Data using Map Reduce
Machine Learning Big Data using Map Reduce By Michael Bowles, PhD Where Does Big Data Come From? -Web data (web logs, click histories) -e-commerce applications (purchase histories) -Retail purchase histories
Advanced Big Data Analytics with R and Hadoop
REVOLUTION ANALYTICS WHITE PAPER Advanced Big Data Analytics with R and Hadoop 'Big Data' Analytics as a Competitive Advantage Big Analytics delivers competitive advantage in two ways compared to the traditional
Matrix Algebra in R A Minimal Introduction
A Minimal Introduction James H. Steiger Department of Psychology and Human Development Vanderbilt University Regression Modeling, 2009 1 Defining a Matrix in R Entering by Columns Entering by Rows Entering
MapReduce and Distributed Data Analysis. Sergei Vassilvitskii Google Research
MapReduce and Distributed Data Analysis Google Research 1 Dealing With Massive Data 2 2 Dealing With Massive Data Polynomial Memory Sublinear RAM Sketches External Memory Property Testing 3 3 Dealing With
Package tagcloud. R topics documented: July 3, 2015
Package tagcloud July 3, 2015 Type Package Title Tag Clouds Version 0.6 Date 2015-07-02 Author January Weiner Maintainer January Weiner Description Generating Tag and Word Clouds.
1 2 3 1 1 2 x = + x 2 + x 4 1 0 1
(d) If the vector b is the sum of the four columns of A, write down the complete solution to Ax = b. 1 2 3 1 1 2 x = + x 2 + x 4 1 0 0 1 0 1 2. (11 points) This problem finds the curve y = C + D 2 t which
Big Data Processing with Google s MapReduce. Alexandru Costan
1 Big Data Processing with Google s MapReduce Alexandru Costan Outline Motivation MapReduce programming model Examples MapReduce system architecture Limitations Extensions 2 Motivation Big Data @Google:
Programming Languages & Tools
4 Programming Languages & Tools Almost any programming language one is familiar with can be used for computational work (despite the fact that some people believe strongly that their own favorite programming
Oracle Data Miner (Extension of SQL Developer 4.0)
An Oracle White Paper September 2013 Oracle Data Miner (Extension of SQL Developer 4.0) Integrate Oracle R Enterprise Mining Algorithms into a workflow using the SQL Query node Denny Wong Oracle Data Mining
MONTE-CARLO SIMULATION OF AMERICAN OPTIONS WITH GPUS. Julien Demouth, NVIDIA
MONTE-CARLO SIMULATION OF AMERICAN OPTIONS WITH GPUS Julien Demouth, NVIDIA STAC-A2 BENCHMARK STAC-A2 Benchmark Developed by banks Macro and micro, performance and accuracy Pricing and Greeks for American
THE NAS KERNEL BENCHMARK PROGRAM
THE NAS KERNEL BENCHMARK PROGRAM David H. Bailey and John T. Barton Numerical Aerodynamic Simulations Systems Division NASA Ames Research Center June 13, 1986 SUMMARY A benchmark test program that measures
Notes on Factoring. MA 206 Kurt Bryan
The General Approach Notes on Factoring MA 26 Kurt Bryan Suppose I hand you n, a 2 digit integer and tell you that n is composite, with smallest prime factor around 5 digits. Finding a nontrivial factor
Information Processing, Big Data, and the Cloud
Information Processing, Big Data, and the Cloud James Horey Computational Sciences & Engineering Oak Ridge National Laboratory Fall Creek Falls 2010 Information Processing Systems Model Parameters Data-intensive
Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware
Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware Created by Doug Cutting and Mike Carafella in 2005. Cutting named the program after
Lecture 6: Introduction to MPI programming. Lecture 6: Introduction to MPI programming p. 1
Lecture 6: Introduction to MPI programming Lecture 6: Introduction to MPI programming p. 1 MPI (message passing interface) MPI is a library standard for programming distributed memory MPI implementation(s)
Package neuralnet. February 20, 2015
Type Package Title Training of neural networks Version 1.32 Date 2012-09-19 Package neuralnet February 20, 2015 Author Stefan Fritsch, Frauke Guenther , following earlier work
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +
Medical Information Management & Mining. You Chen Jan,15, 2013 [email protected]
Medical Information Management & Mining You Chen Jan,15, 2013 [email protected] 1 Trees Building Materials Trees cannot be used to build a house directly. How can we transform trees to building materials?
Big Data Analytics with Spark and Oscar BAO. Tamas Jambor, Lead Data Scientist at Massive Analytic
Big Data Analytics with Spark and Oscar BAO Tamas Jambor, Lead Data Scientist at Massive Analytic About me Building a scalable Machine Learning platform at MA Worked in Big Data and Data Science in the
Hadoop and Map-Reduce. Swati Gore
Hadoop and Map-Reduce Swati Gore Contents Why Hadoop? Hadoop Overview Hadoop Architecture Working Description Fault Tolerance Limitations Why Map-Reduce not MPI Distributed sort Why Hadoop? Existing Data
Compute Cluster Server Lab 3: Debugging the parallel MPI programs in Microsoft Visual Studio 2005
Compute Cluster Server Lab 3: Debugging the parallel MPI programs in Microsoft Visual Studio 2005 Compute Cluster Server Lab 3: Debugging the parallel MPI programs in Microsoft Visual Studio 2005... 1
Risk Decomposition of Investment Portfolios. Dan dibartolomeo Northfield Webinar January 2014
Risk Decomposition of Investment Portfolios Dan dibartolomeo Northfield Webinar January 2014 Main Concepts for Today Investment practitioners rely on a decomposition of portfolio risk into factors to guide
Linux Cluster Computing An Administrator s Perspective
Linux Cluster Computing An Administrator s Perspective Robert Whitinger Traques LLC and High Performance Computing Center East Tennessee State University : http://lxer.com/pub/self2015_clusters.pdf 2015-Jun-14
Octave and Python: High-Level Scripting Languages Productivity and Performance Evaluation
Octave and Python: High-Level Scripting Languages Productivity and Performance Evaluation Juan Carlos Chaves, John Nehrbass, Brian Guilfoos, Judy Gardiner, Stanley Ahalt, Ashok Krishnamurthy, Jose Unpingco,
CDD user guide. PsN 4.4.8. Revised 2015-02-23
CDD user guide PsN 4.4.8 Revised 2015-02-23 1 Introduction The Case Deletions Diagnostics (CDD) algorithm is a tool primarily used to identify influential components of the dataset, usually individuals.
Your Best Next Business Solution Big Data In R 24/3/2010
Your Best Next Business Solution Big Data In R 24/3/2010 Big Data In R R Works on RAM Causing Scalability issues Maximum length of an object is 2^31-1 Some packages developed to help overcome this problem
High-Volume Data Warehousing in Centerprise. Product Datasheet
High-Volume Data Warehousing in Centerprise Product Datasheet Table of Contents Overview 3 Data Complexity 3 Data Quality 3 Speed and Scalability 3 Centerprise Data Warehouse Features 4 ETL in a Unified
Oracle Advanced Analytics 12c & SQLDEV/Oracle Data Miner 4.0 New Features
Oracle Advanced Analytics 12c & SQLDEV/Oracle Data Miner 4.0 New Features Charlie Berger, MS Eng, MBA Sr. Director Product Management, Data Mining and Advanced Analytics [email protected] www.twitter.com/charliedatamine
SmartArrays and Java Frequently Asked Questions
SmartArrays and Java Frequently Asked Questions What are SmartArrays? A SmartArray is an intelligent multidimensional array of data. Intelligent means that it has built-in knowledge of how to perform operations
Variance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers
Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications
High Throughput Computing on P2P Networks. Carlos Pérez Miguel [email protected]
High Throughput Computing on P2P Networks Carlos Pérez Miguel [email protected] Overview High Throughput Computing Motivation All things distributed: Peer-to-peer Non structured overlays Structured
HPC ABDS: The Case for an Integrating Apache Big Data Stack
HPC ABDS: The Case for an Integrating Apache Big Data Stack with HPC 1st JTC 1 SGBD Meeting SDSC San Diego March 19 2014 Judy Qiu Shantenu Jha (Rutgers) Geoffrey Fox [email protected] http://www.infomall.org
The Image Deblurring Problem
page 1 Chapter 1 The Image Deblurring Problem You cannot depend on your eyes when your imagination is out of focus. Mark Twain When we use a camera, we want the recorded image to be a faithful representation
STATISTICA Solutions for Financial Risk Management Management and Validated Compliance Solutions for the Banking Industry (Basel II)
STATISTICA Solutions for Financial Risk Management Management and Validated Compliance Solutions for the Banking Industry (Basel II) With the New Basel Capital Accord of 2001 (BASEL II) the banking industry
DATA ANALYSIS II. Matrix Algorithms
DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where