PivotalR: A Package for Machine Learning on Big Data

Size: px

Start display at page:

Download "PivotalR: A Package for Machine Learning on Big Data"

Kenneth Hood
8 years ago
Views:

2 What Can Small Data Scientists Bring on Their Big Data Journey? Small Data Ra nd p use ython r Databases In-me m Flat files ory m buildin odel g Command-line tools Big Data nd Ra n o pyth r use MapRe duce Many tools and approaches are being adapted to big data technologies S HDF Cloud computing P MP ses aba t a d puting m o c uted b i r t s i D Command-line tools Copyright 2014 Pivotal. All rights reserved.

Big Data nd Ra n o pyth r use MapRe duce Many tools and approaches are being adapted to big data

Tools for Data Scientists COMMERCIAL OPEN SOURCE (OR FREE)

PivotalR Design Overview Database/Hadoop w/ MADlib RPostgreSQL PivotalR 2. SQL to execute 1.

5 Data Operators crossprod, scale, sample Arith, Logical, Cast Extraction: x[,-2], x[,1:3], x[,c( rings, sex )], x$arr[, 1] Replacement: x[x$sex == I,] <- NA Support array columns Easy to construct complicated SQL queries. For example, filter NULL values for (i in 1:ncol(w)) w <- w[!is.na(w[i]),] Copyright 2013 Pivotal. All rights reserved.

array columns Easy to construct complicated SQL queries.

6 Machine learning Current MADlib wrapper functions: madlib.lm, madlib.glm, madlib.summary, madlib.arima, madlib.elnet Related functions: generic.cv, generic.bagging, margins, predict Support for formula y Support for categorical variables as.factor, predict Copyright 2013 Pivotal. All rights reserved. ~. - x[1:2] - z + factor(w) relevel,

bagging, margins, predict Support for formula y Support for categorical variables as.

7 MADlib In-database Machine Learning Library Pivotal Confidential Internal Use Only

How Pivotal Data Scientists Select Which Pivotal

MADlib: Toolkit for Advanced Big Data Analytics Better Parallelism Algorithms designed to leverage MPP or Hadoop architecture Better Scalability Algorithms scale as your data set scales No data

9 MADlib: Toolkit for Advanced Big Data Analytics Better Parallelism Algorithms designed to leverage MPP or Hadoop architecture Better Scalability Algorithms scale as your data set scales No data movement Better Predictive Accuracy Using all data, not a sample, may improve accuracy Open Source Available for customization and optimization by user Copyright 2014 Pivotal. All rights reserved.

No data movement Better Predictive Accuracy Using all data, not a sample, may improve accuracy

10 Which platforms does it run on? (Partly ported) Impala HAWQ HDFS Pivotal Confidential Internal Use Only GPDB PostgreSQL

11 Shared-Nothing Database Architecture MPP (Massively Parallel Processing) Master Servers... SQL MapReduce... Query planning & dispatch Network Interconnect Segment Servers Query processing & data storage External Sources Loading, streaming, etc. Pivotal Confidential Internal Use Only......

.. Query planning & dispatch Network Interconnect Segment Servers Query

12 Pivotal Confidential Internal Use Only

13 Example usage Train a model Predict for new data Pivotal Confidential Internal Use Only

14 But not all Data Scientists speak SQL Accessing Scalability through R Pivotal Confidential Internal Use Only

15 PivotalR: A familiar R interface Current version Pivotal R d <- db.data.frame( houses") houses_linregr <- madlib.lm( price ~ tax + bath + size, data=d) Copyright 2014 Pivotal. All rights reserved. SQL Code SELECT madlib.linregr_train( 'houses, 'houses_linregr, 'price, 'ARRAY[1, tax, bath, size] );

lm( price ~ tax + bath + size, data=d) Copyright 2014 Pivotal.

cv margins Not easy to do in MADlib on the

17 Quick Prototype Examples (see the script): Linear regression PCA Poisson regression Left inverse of a matrix AdaBoost Portable Same code on all supported platforms Copyright 2013 Pivotal. All rights reserved.

matrix AdaBoost Portable Same code on all supported

Testing Framework R CMD INSTALL --install-tests PivotalR_0.1.16.2.

Enabling R for Big Data with PL/R and PivotalR Real World Examples on Hadoop & MPP Databases

Enabling R for Big Data with PL/R and PivotalR Real World Examples on Hadoop & MPP Databases Woo J. Jung Principal Data Scientist Pivotal Labs 1 All In On Open Source Still can t believe we did this. Truly