Big Data, beating the Skills Gap Using R with Hadoop

Size: px

Start display at page:

Download "Big Data, beating the Skills Gap Using R with Hadoop"

Joshua Kelley
8 years ago
Views:

1 Big Data, beating the Skills Gap Using R with Hadoop

2 Using R with Hadoop There are a number of R packages available that can interact with Hadoop, including: hive - Not to be confused with Apache Hive, which is effectively SQL for Hadoop HadoopStreaming - Provides a framework for writing map/reduce scripts for use in Hadoop Streaming RHadoop - A collection of five R packages that allow users to manage and analyze data with Hadoop khanley@mango-solutionscom

HadoopStreaming - Provides a framework for writing map/reduce scripts for use in Hadoop Streaming

3 What do I need to know about Hadoop to understand this talk? For the purposes of this talk, we need to know about the following: HDFS MapReduce khanley@mango-solutionscom

4 HDFS What is it? This stands for Hadoop Distributed File System A way of storing data across multiple machines to allow ease of access The data is broken up into chunks and spread across multiple machines Why bother? No massive files which contain terabytes of information There are multiple copies of each file dotted around different machines, so if one of your machines go down or the data is corrupted, you can still perform the analysis khanley@mango-solutionscom

access The data is broken up into chunks and spread across multiple machines Why bother?

5 MapReduce What is it? A way of coding your problem so that it can be split up and spread across multiple machines Each machine performs its own part of the analysis, and the results are then collected together at the end khanley@mango-solutionscom

7 RHadoop We ll focus on using some of the RHadoop packages, as these seem to be widely used Not on CRAN, but available to download on github RHadoop is divided into a number of packages We re going to use the rmr2 package, which allows you to use R to perform MapReduce khanley@mango-solutionscom

RHadoop is divided into a number of packages We re going to use the rmr2

8 A simple MapReduce task using R The task - Perform a word count on the text of James Joyce s Ulysses Input - A text file containing the plain text version of the novel Output - An R object containing all unique words and a count specifying the number of times it occurred in the novel Simplifications - We removed all punctuation from the file before beginning the analysis This avoids issues where hello and hello! are treated as different words khanley@mango-solutionscom

specifying the number of times it occurred in the novel Simplifications - We removed all punctuation from the file

9 A simple MapReduce task using R Step 1: Load up the rmr2 package library(rmr2) khanley@mango-solutionscom

10 A simple MapReduce task using R Option 2a: You can try out the package without having Hadoop installed This will allow you to play around with the package, but you won t be able to execute any Hadoop jobs To enable the package to work locally, you need to set the following option: rmroptions(backend="local") khanley@mango-solutionscom

you won t be able to execute any Hadoop jobs To enable the package to work locally,

11 A simple MapReduce task using R Option 2b: If you have a Hadoop installation available, you need to tell the package where to locate it The exact paths will vary depending on your Hadoop installation (ask your IT department!), but for example, on our system the options were set as follows: Syssetenv(HADOOP_CMD = "/usr/local/hadoop/bin/hadoop") Syssetenv(HADOOP_STREAMING = "/usr/local/hadoop/share/ hadoop/tools/lib/hadoop-streaming-241jar") Syssetenv(JAVA_HOME = "/usr/lib/jvm/java") khanley@mango-solutionscom

), but for example, on our system the options were set as follows: Syssetenv(HADOOP_CMD = "/usr/local/hadoop/bin/hadoop")

12 A simple MapReduce task using R Step 3: Tell R where it can set up a temporary directory for storing data (you will need to have write permission here!) rmroptions(hdfstempdir = "/user/kate") khanley@mango-solutionscom

(you will need to have write permission here!

13 A simple MapReduce task using R Step 4: Store the data in the distributed file system The data is not held in memory, but dfsuly points to its location dfsuly <- todfs(readlines("pg4300txt")) khanley@mango-solutionscom

memory, but dfsuly points to its location dfsuly <-

14 A simple MapReduce task using R Step 5: Write a map function The map function expects to receive a single line of the file at a time and creates key-value pairs for each word in the file For this example, the key will be the word that we want to include in our total, and the value will always be 1 At the end, we will sum up all the 1 s for each word/key, which will give us the number of times it appeared in the book khanley@mango-solutionscom

the word that we want to include in our total, and the value will always be 1 At the end, we will sum up all

15 A simple MapReduce task using R Step 5: Write a map function wcmap <- function(, lines) { # Splits the line into a vector of individual words vecofwords <- strsplit(x = lines, split = " ")[[1]] } # Creates a key-value pair for each word, # assigning them the value 1 keyval(vecofwords, 1) khanley@mango-solutionscom

vecofwords <- strsplit(x = lines, split = " ")[[1]] } # Creates a key-value

16 A simple MapReduce task using R Step 6: Write a reduce function wcreduce <- function(word, counts) { # For each word, sum the counts keyval(word, sum(counts)) } khanley@mango-solutionscom

17 A simple MapReduce task using R Step 7: Run your code and view the results Use the MapReduce function to run your analysis res <- mapreduce( input = dfsuly, map = wcmap, reduce = wcreduce ) # I like to convert the results into a # data frame for ease of use resultsdfs <- fromdfs(res) results <- asdataframe(resultsdfs) khanley@mango-solutionscom

reduce = wcreduce ) # I like to convert the results into a # data frame for ease of

18 The Results! We can then take a look at the results: head(results) ## key val ## 1 hoof 9 ## 2 hook 10 ## 3 hoop 3 ## 4 hoot 1 ## 5 hope 64 ## 6 hopk 1 khanley@mango-solutionscom

19 The Results! Most common word - the (14,932 times) Most common word with more than 5 letters - stephen (505 times) Most common word with more than 10 letters - shakespeare (39 times) Total number of words - 267,175 khanley@mango-solutionscom

20 To Conclude R and Hadoop work really well together There are plenty of packages out there that allow you to use MapReduce with R Unless you re very tech-savvy, you may well need support (either from your IT team, or externally) to get Hadoop up and running khanley@mango-solutionscom

re very tech-savvy, you may well need support (either from your IT

Package hive. January 10, 2011

Package hive. January 10, 2011 Package hive January 10, 2011 Version 0.1-9 Date 2011-01-09 Title Hadoop InteractiVE Description Hadoop InteractiVE, is an R extension facilitating distributed computing via the MapReduce paradigm. It