MapReduce Laboratory

Transcription

1 MapReduce Labratry In this labratry students will learn hw t use the Hadp client API by wrking n a series f exercises: The classic Wrd Cunt and variatins n the theme Design Pattern: Pair and Stripes Design Pattern: Order Inversin Jin implementatins Nte that the design patterns utlined abve have been riginally discussed in: Jimmy Lin, Chris Dyer, Data-Intensive Text Prcessing with MapReduce, Mrgan Claypl 1 Run the Virtual Machine 1.1 VM details It is precnfigured with all the sftware needed t get thrugh the assignments. It has an Ubuntu LTE (64bit) with XFCE Desktp Envirnment It requires 2GB f memry It has a single accunt: username: student passwrd: passwrd Ntice that lgin is passwrd less and rt cmmands can be run using sud which is passwrd less as well. IMPORTANT: Shut dwn the VM nce finished the sessin (d nt clse abruptly the windw) 1.2 Run the VM n the laptp Dwnlad the image tarball frm (r ask t the teacher the USB key) and uncmpress it The image it can be used either VMplayer r Virtualbx VMplayer: Windws/Linux: dwnlad the freeware player frm here Mac OS: unfrtunately there is n freeware player, but it can be used VMware Fusin which has a 30-day trial. Virtualbx: The image can be laded in Virtualbx with n issues. Be sure t select the file HadpVM.vmdk as a virtual hard-drive (and nt ne f the HadpVM-sXXX.vmdk files). T install guest additins, nce the image is running: sud apt-get install dkms build-essential dwnlad the additins (devices! install guest additins) sud su - cd /media/virtualboxaddition_versin./vbxlinuxadditin.run 1.3 Run the VM n the labratry machines The VM image is stred in a lcal directry. This means that: The student needs t use every time the same physical machine; any mdificatin will affect the VM n that specific physical machine; The physical machine and the lcal directry can be accessed by anyne;

2 T prtect the access t the VM, nce the VM is up and running, the student may add a lgin screen (remember t change the passwrd f the user student ). This may nt prevent ther peple frm deleting the whle VM frm the machine, therefre remember t save the wrk smewhere utside the VM (e.g., backup f the mdified Java files using a shared directry, which can be inside the student user space). The VM is accessible frm the user menu. It can be fund in Menu! Istruzine! Virtual Machine Hadp IMPORTANT: Shut dwn the VM befre leaving (d nt clse abruptly the windw) 2 Setup the labratry surces in Eclipse First, pen the brwser and g t the curse web page, where yu can find the file with the surce cde that will be used in this lab sessin. Dwnlad the surce cde and uncmpress it in any f yur lcal directry n the VM. Then, create a new Java Prject in Eclipse: Inside Eclipse select the menu item File > New > Prject... t pen the New Prject wizard. Select Java Prject then click Next t start the New Java Prject Type a name fr yur new prject, such as ''mr-lab'' Ensure t use JavaSE-1.6 as JRE, then click n Next G n Libraries tab, and click n Add External Jars Add the fllwing jars: /usr/lib/hadp/hadp-cmmn.jar /usr/lib/hadp-0.20-mapreduce/hadp-cre.jar /usr/lib/hadp-hdfs/hadp-hdfs.jar Click n Finish Nw yu have a new prject. The next step is t imprt the surce files t cmplete in the Prject. Inside Eclipse, select the src directry in yur prject. Right-click n the src directry and select Imprt Select General / File System then click Next In Frm Directry, select./mr/src/, which is inside the directry where yu clned the git repsitry In the tree n the left, expand src and select (put the tick n) fr, then click Finish. At this pint yu shuld have a java prject named mapred-lab already cnfigured. The next step is starting with the first exercise. Nte that there can be errrs in the surce cde yu imprted. This is nrmal, since there are uncmpleted surce files that will be crrected in the exercises. 3 Utilities 3.1 Hw t launch a jb In rder t launch a jb, yu need first t exprt a JAR file. Therefre, frm Eclipse: Select the menu item File > Exprt Type JAR, select JAR file and click n Next In the textbx Select the exprt destinatin, type the name f the new JAR yu want t exprt, i.e. '/hme/student/mrlab.jar' Type Finish. Once yu have exprted yur jar file, yu can run yur cde using the lcal versin f Hadp. Open a terminal, and type: hadp jar <jarname.jar> <fully.qualified.class.name> <Parameters> Fr example, fr running the WrdCunt exercise, type:

3 hadp jar mrlab.jar fr.eurecm.dsg.mapreduce.wrdcunt 2 INPUT/text/qute.txt OUTPUT/wrdcunt/ Nte that yu need t specify a nn existing utput directry, r t delete it befre running the jb. 3.2 Navigate HDFS The HDFS client can talk t the namende t btain infrmatin n the distributed file system, and t make a number f peratins. The basic cmmand is hadp fs [ptin] where ptin can be ls R! Displays recursively the cntent f the file system -get <src> <lcaldst>! Cpies files frm hdfs t the lcal file system cat <file>! Displays <file> n the terminal (cpies t stdut) and many thers see the Hadp FS shell cmmands fficial dcumentatin. Alternatively, ne can use the web interface t interact with the namende (see next sectin). 3.3 Web interfaces: Mnitr jb prgress Hadp publishes sme web interfaces that display JbTracker and HDFS statuses. Yu can access them using the fllwing links: Jbtracker Web Interface: NameNde Web Interface: 4 Exercises Nte, exercises are rganized in ascending rder f difficulty. 4.1 Exercise 1 Wrd Cunt Cunt the ccurrences f each wrd in a text file. This is the simplest example f MapReduce jb: in the fllwing we illustrate three pssible implementatins. Basic: the map methd emits 1 fr each wrd. The reduce aggregates the nes it receives frm mappers fr each key it is respnsible fr and save n disk (HDFS) the result In-Memry Cmbiner: instead f emitting 1 fr each encuntered wrd, the map methd (partially) aggregates the nes and emit the result fr each wrd Cmbiner: the same as In-Memry Cmbiner but using the MapReduce Cmbiner class Fr the basic versin the student has t mdify the file WrdCunt.java in the package fr.eurecm.dsg.mapreduce. The user must perate n each TODO filling the gaps with cde, fllwing the descriptin assciated t each TODO. The package java.util cntains a class StringTkenizer that can be used t tkenize the text. Fr the In-Memry Cmbiner and the Cmbiner the student has t mdify WrdCuntIMC.java and WrdCuntCmbiner.java in the same package referenced abve. The student has t perate n each TODO using the same cde f the basic wrd cunt example except fr the TODOs marked with a star *. Thse must be cmpleted with using the apprpriate design pattern. When an exercise is cmpleted yu can exprt it int a jar file that will then be used t execute it.

4 The final versin shuld get in input three arguments: the number f reducers, the input file and the utput path. Example f executin are: hadp jar <cmpiled_jar> fr.eurecm.dsg.mapreduce.wrdcunt 3 <input_file> <utput_path> hadp jar <cmpiled_jar> fr.eurecm.dsg.mapreduce.wrdcuntimc 3 <input_file> <utput_path> hadp jar <cmpiled_jar> fr.eurecm.dsg.mapreduce.wrdcuntcmbiner 3 <input_file> <utput_path> T test yur cde use the file /user/student/input/text/qute.txt, saved in the lcal HDFS fs. T run the final versin f yur jb, yu can use a bigger file, /user/student/input/text/gutenbergpartial.txt, which cntains an extract f the English bks frm Prject Gutenberg ( which prvides a cllectin f full texts f public dmain bks. 4.2 Exercise 2 Term c-ccurrences In the fllwing exercise, we need t build the term c-ccurrence matrix fr a text cllectin. A c-ccurrence matrix is a n x n matrix, where n is the number f unique wrds in the text. Fr each cuple f wrds, we cunt the number f times they c-ccurred in the text in the same line Pairs Design Pattern The basic (and maybe mst intuitive) implementatin f this exercise is the Pair. The basic idea is t emit, fr each cuple f wrds in the same line, the cuple itself (r pair) and the value 1. Fr example, in the line w1 w2 w3 w1, we emit (w1,w2):1, (w1,w3):1, (w2,w1):1, (w2,w3):1, (w2,w1):1, (w3,w1):1, (w3,w2):1, (w3,w1):1. In this exercise, we need t use a cmpsite key t emit an ccurrence f a pair f wrds. The student will understand hw t create a custm Hadp data type t be used as key type. A Pair is a tuple cmpsed by tw elements that can be used t ship tw bjects within a parent bject. Fr this exercise the student has t implement a TextPair, that is a Pair that cntains tw wrds. There are tw files fr this exercise: TextPair.java: data structure t be implemented by the student. Besides the implementatin f the data structure itself, the student has t implement the serializatin Hadp API (write and read Fields). Pair.java: the implementatin f a pair example using TextPair.java as datatype. The final versin shuld get in input three arguments: the number f reducers, the input file and the utput path. Example f executin are: hadp jar <cmpiled_jar> fr.eurecm.dsg.mapreduce.pair 1 <input_file> <utput_path> T test yur cde use the file /user/student/input/text/qute.txt, saved in the lcal HDFS fs. T run the final versin f yur jb, yu can use a bigger file, /user/student/input/text/gutenbergpartial.txt.

5 4.2.2 Stripes Design Pattern This apprach is similar t the previus ne: fr each line, c-ccurring pairs are generated. Hwever, nw, instead f emitting every pair as sn as it is generated, intermediate results are stred in an assciative array. We use an assciative array, and, fr each wrd, we emit the wrd itself as key and a Stripe, that is the map f c-ccurring wrds with the number f assciated ccurrence. Fr example, in the line w1 w2 w3 w1, we emit: w1:{w2:1, w3:1}, w2:{w1:2,w3:1}, w3:{w1:2, w2:1}, w1:{w2:1, w3:1} Nte that, instead, we culd emit als: w1:{w2:2, w3:2}, w2:{w1:2,w3:1}, w3:{w1:2, w2:1} In this exercise the student will understand hw t create a custm Hadp data type t be used as value type. There are tw files fr this exercise: StringTIntMapWritable.java: the data structure file, t be implemented Stripes.java: the MapReduce jb, that the student must implement using the StringTIntMapWritable data structure hadp jar <cmpiled_jar> fr.eurecm.dsg.mapreduce.stripes 2 <input_file> <utput_path> T test yur cde use the file /user/student/input/text/qute.txt, saved in the lcal HDFS fs. T run the final versin f yur jb, yu can use a bigger file, /user/student/input/text/gutenbergpartial.txt. 4.3 Exercise 3 Relative term c-ccurrence and Order Inversin Design Pattern In this example we need t cmpute the c-ccurrence matrix, like the ne in the previus exercise, but using the relative frequencies f each pair, instead f the abslute value. Pratically, we need t cunt the number f times each pair (w i, w j ) ccurs divided by the number f ttal pairs with w i (marginal). The student has t implement the Map and Reduce methds and the special partitiner (see OrderInversin#PartitinerTextPair class), which apply the partitiner nly accrding t the first element in the Pair, sending all data regarding the same wrd t the same reducer. Nte that inside the OrderInversin class there is a field called ASTERISK which shuld be used t utput the ttal number f ccurrences f a wrd. There is ne file fr this exercise called OrderInversin.java. The run methd f the jb is already implemented, the student shuld cmplete the mapper, the reducer and the partitiner, as explained in the TODOs. hadp jar <cmpiled_jar> fr.eurecm.fr.mapreduce.orderinversin 4 <input_file> <utput_path> T test yur cde use the file /user/student/input/text/qute.txt, saved in the lcal HDFS fs.

6 T run the final versin f yur jb, yu can use a bigger file, /user/student/input/text/gutenbergpartial.txt. 4.4 Exercise 4 Jins In MapReduce the term jin refers t merging tw different dataset stred as unstructured files in HDFS. As fr databases, in MapReduce there are many different kind f jins, each with its use-cases and cnstraints. In this labratry the student will implement the Reduce-Side Jin: the map phase tags each recrd such that recrds f different inputs that have t be jined will have the same tag. Each reducer will receive a tag with a list f recrds and perfrm the jin. Jbs Reduce Side Jin: Yu need t find the tw-hps friends, i.e. the friends f friends f each user, in the twitter dataset. In particular, yu need t implement a self-jin, that is a jin between tw instances f the same dataset, n the twitter graph. T test yur cde use the file /user/student/input/twitter/twitter-small.txt, saved in the lcal HDFS fs. T run the final versin f yur jb, yu can use a bigger file, /user/student/input/twitter/twitter-bigsample.txt. Bth files cntain lines in the frm userid friendid. Reduce Side Jin: use the file ReduceSideJin.java as starting pint. This exercise is different frm the thers because it des nt cntain any infrmatin n hw t d it. The student is free t chse hw t implement it. hadp jar <cmpiled_jar> fr.eurecm.fr.mapreduce.reducesidejin 1 <input_file2> <input_file1> <utput_path> 4.5 Exercise 5 PageRank In this example, we cmpute the simple versin f the PageRank, i.e., we d nt cnsider the jump factr and the presence f sink ndes (every nde has at least ne utput link, and the graph is cmpletely cnnected). The graph is described thrugh its adjacency list: in particular, the starting input file has the fllwing frmat fr each line: ndeid currentpagerank neighndeid1 neighndeid2 neighndeid3... where ndeid is the unique identifier f the nde, currentpagerank is the current value f the PageRank (initially set t the inverse f the number f ndes), and then there are the identifiers f the utput links (the graph is directed, i.e., the adjacency matrix is nt symmetric). The files f tw different graphs can be dwnladed frm the curse webpage (and shuld be stred in HDFS). The student has t implement the Map and Reduce methds fr the simplified PageRank cmputatin. The run methd f the jb shuld launch a finite number f iteratins (given as input parameter). At each iteratin, the intermediate directry shuld be remved, s that, at the end, there will be nly the initial directry and the final ne. Nte that, since the utput f a MapReduce cmputatin is a directry, and the PageRank cmputatin is iterative, then the input (even at the beginning) shuld be always a directry. The student, therefre, shuld put the files describing a graph in separated directries.