Big Data 2012 Hadoop Tutorial

Transcription

1 Big Data 2012 Hadoop Tutorial Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 1

2 Contact Exercise Session Friday to CHN D 46 Your Assistant Martin Kaufmann Office: CAB E martinka@inf.ethz.ch Download of Exercises: /fall2012/bigdata Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 2

3 MapReduce Parallel problems distributed across huge data sets using a large number of nodes Two stages: Map step: master node takes input, divides into smaller sub-problems Reduce step: master node collects answer from all sub-problems and combindes them in some way Condition: reduction function is associative Remember: A x (B x C) = (A x B) x C Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 3

4 MapReduce MapReduce transforms (key, value) pairs into list of values: Map and Reduce functions defined with respect data stored in KV pairs: Map(k1, v2) list(k2,v2) MapReduce then groups all pairs with same key Reduce(k2, list(v2)) list(v3) All functions executed in parallel! Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 4

5 Dataflow of MapReduce Input reader: divides input into splits. One split is assigned to a map function. Map function: Takes (key, value) pairs and generates one or more output KV pairs Partition function: Asigning each map function to a reducer. Returns an index of reduce. Comparison function: The input for each reduce is sorted using a comparison function. Reduce function: The reduce function is called once for each unique key in the sorted order iterating through values and producing zero, one or more outputs Output writer: writes output of reduce to storage Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 5

6 Overview of Hadoop provides a programming model Efficient, automatic distribution of data & work across machines Open source implementation of Google s MapReduce FW on top of Hadoop Distributed File System (HDFS) Large-scale distributed batch processing for vasts amount of data (multi-terabytes) Runs on large clusters (1000s of nodes) of commodity hw with reliability & fault-tolerance Highly scalable filesystem, computing coupled to storage Provides a simplified programming model: map() & reduce() no schema or type support Slides adopted by Cagri Balkesen Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 6

7 Namenode: HDFS master server Manages the filesystem namespace (block mappings) Regulates access to files by clients (open, close, rename,...) Datanode: Manages data attached to each node Data is split into blocks & replicated (default is 64MB) Serves r/w requests of blocks Data locality, computing goes to data effective scheduling & parallel processing High aggregate bandwidth HDFS Architecture Image Sources: [1] [2] Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 7

8 The MapReduce Paradigm Mapping Lists MAP SHUFFLE SORT Reducing Lists REDUCE Maps execute in parallel over different local chunks Map outputs shuffled/copied to reduce nodes Reduce tasks begin after all local data is sorted Image Sources: [1] Mapper(filename,contents): for each word in contents emit(word, 1) Reducer(word, values): sum = 0 for each value in values: sum = sum + value emit(word, sum) WordCount Example Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 8

9 MapReduce Terminology Job: A «full program» an execution of a Mapper and Reducer across a data set Task: An execution of a Mapper or a Reducer on a slice of data. a.k.a Task-In-Progress (TIP) Master node runs JobTracker instance, which accepts Job requests from clients TaskTracker instances run on slave nodes, periodically query JobTracker for work TaskTracker forks separate Java process for task instances, failures isolated & restarts with same input All mappers are equivalent; so map whatever data is local to a particular node in HDFS TaskRunner launches Mapper/Reducer & knows which InputSplits should be processed; calls Mapper/Reducer for each record from the InputSplit Ex: InputSplit each 64MB file chunk; RecordReader each line in chunk, also InputFormat identifies the InputSplit(i.e. TextInputFormat) Partitioner: Used in shuffle & determines the partition number for a key Credits: [3] Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 9

10 The WordCount Example function map(string name, String document): // name: document name // document: document contents for each word w in document: emit (w, 1) function reduce(string word, Iterator partialcounts): // word: a word // partialcounts: a list of aggregated partial counts sum = 0 for each pc in partialcounts: sum += pc emit (word, sum) Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 10

11 The WordCount Example Dataset: Splits: We are not what we want to be, but at least we are not what we used to be. InputSplit-1 InputSplit-2 InputSplit-3 InputSplit-4 (k1,v1) (k2,v2) (k3,v3) (k4,v4) (k5,v5) We are not what we want to be, but at least we are not what we used to be. InputSplits are read and processed via TextInputFormat Parses input Generates key-value pairs: (key=offset, value=line-contents) InputSplit boundaries expanded to newline \n Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 11

12 The WordCount Example (k1,v1) (k2,v2) (k3,v3) (k4,v4) (k5,v5) We are not what we want to be, but at least we are not what we used to be. Map(k1,v1) Map(k2,v2) <we, 1> <are, 1> <not, 1> <what, 1> <we, 1> <want, 1> <to, 1> <be, 1> Map(k3,v3) <but, 1> <at, 1> <least, 1> Map(k4,v4) <we, 1> <are, 1> <not, 1> <what, 1> Map(k5,v5) <we, 1> <used, 1> <to, 1> <be, 1> <we, 4> Reduce(k,v[]) <we, 1> <we, 1> <we, 1> <we, 1> <are, 2> <not, 2> Reduce(k,v[]) Reduce(k,v[]) <are, 1> <are, 1> <not, 1> <not, 1> <what, 2> Reduce(k,v[]) <what, 1> <what, 1> Shuffle/Sort Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 12

13 Setting up Hadoop 3 modes of setup Standalone: Single Java process, for verification & debugging Pseudo-distributed: Single machine, but JobTracker & NameNode on different processes Fully-distributed: JobTracker & NameNode on different machines together with other slave machines Let s try standalone: Download the latest stable release: Extract files: tar xvz hadoop-1.0.4*.tar.gz Set the following in conf/hadoop-env.sh JAVA_HOME=/usr/java/default In Hadoop directory Create an input folder: $~/hadoop> mkdir input Download & extract the sample: $~/hadoop/input> wget 0.gz Run the word count example $~/hadoop> bin/hadoop jar hadoop-examples-*.jar wordcount input/ out/ See the results in out/ Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 13

14 Dissecting the Word Count code The source code of Word Count is src/examples/org/apache/hadoop/examples/wordcount.java Mapper class: public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasmoretokens()) { word.set(itr.nexttoken()); context.write(word, one); } } } Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 14

15 Reducer class: Dissecting the Word Count code public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 15

16 Job setup: Dissecting the Word Count code public static void main(string[] args) throws Exception { Configuration conf = new Configuration(); String[] otherargs = new GenericOptionsParser(conf, args).getremainingargs(); if (otherargs.length!= 2) { System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setjarbyclass(wordcount.class); job.setmapperclass(tokenizermapper.class); job.setcombinerclass(intsumreducer.class); job.setreducerclass(intsumreducer.class); job.setoutputkeyclass(text.class); job.setoutputvalueclass(intwritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true)? 0 : 1); } Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 16

17 Pseudo-distributed Setup Hadoop still runs in a single machine but simulates distributed setup by different processes for JobTracker & NameNode Change the configuration files as follows: conf/core-site.xml: <configuration> <property> <name> fs.default.name </name> <value> hdfs://localhost:9000 </value> </property> <configuration> conf/hdfs-site.xml: <configuration> <property> <name> dfs.replication </name> <value> 1 </value> </property> </configuration> conf/mapred-site.xml <configuration> <property> <name> mapred.job.tracker </name> <value> localhost:9001 </value> </property> <configuration> Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 17

18 Setup SSH, HDFS and start Hadoop Check whether you can connect without a passphrase: $> ssh localhost If not, setup by executing the following: $> ssh-keygen -t rsa -P '' $> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys Format and make the HDFS ready: $> bin/hadoop namenode -format Start Hadoop daemons: $> bin/start-all.sh Browse the web-interface of NameNode & JobTracker NameNode - JobTracker - Copy input files to the HDFS $> bin/hadoop dfs put localinput dfsinput Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 18

19 Job Tracker Web-Interface Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 19

20 NameNode Web-Interface Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 20

21 HDFS Commands $> ~/hadoop> bin/hadoop dfs [-ls <path>] [-du <path>] [-cp <src> <dst>] [-rm <path>] [-put <localsrc> <dst>] [-copyfromlocal <localsrc> <dst>] [-movefromlocal <localsrc> <dst>] [-get [-crc] <src> <localdst>] [-cat <src>] [-copytolocal [-crc] <src> <locdst>] [-movetolocal [-crc] <src> <locdst>] [-mkdir <path>] [-touchz <path>] [-test -[ezd] <path>] [-stat [format] <path>] [-help [cmd]] Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 21

22 Example Input Data is a sample set of tweets from Twitter as follows (one tweet per line): {``text``:``tweet contents #TAG1 #TAG2``,..., ``hashtags``: [ {``text``:``tag1``,...},..., {``text``:``tag2``,...} ],... } \n Output the tags that occurs more than 10 times in the sample data set along with their occurrence counts. Sample output: TAG1 11 TAG2 50 TAG Implement by modifying from the WordCount.java Compile your source by: (you might need to download Apache Commons CLI first) > cd src/examples > javac -cp../../hadoop-core jar:../../../lib/commonscli-1.2.jar org/apache/hadoop/examples/hashtagfreq.java Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 22

23 Example Input Data is a sampled set of trades on stock market for a single day on 06/01/2006. The contents are as follows: DATE EX TIME PRICE SIZE IBM 06/01/2006 N \n IBM 06/01/2006 N \n SUN 06/01/2006 N \n SUN 06/01/2006 N \n Task: Compute the total volume of trades for each stock ticker and return all stocks having a volume higher than a given value from commandline. In SQL: SELECT symbol, SUM(price*size) AS volume FROM Ticks GROUP BY symbol HAVING volume > V Example total volume for IBM: 84.22* *100 = Sample output: IBM Let s assume filter = 20K IBM SUN Implement by modifying from the WordCount.java Create a directory in your $HADOOP_HOME, let s say stocks/ Copy src/org/apache/hadoop/examples/wordcount.java to stocks/ Modify the code & name accordingly Compile: javac -cp hadoop-core jar:lib/commons-cli-1.2.jar stocks/stockvolume.java Copy dataset to input/ : Run: > bin/hadoop stocks/stockvolume input/ output/ Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 23

24 Setting Job Specific Parameters Set in the main, before submitting the Job: job.getconfiguration().setint("filter", Integer.parseInt(otherArgs[2])); Use inside map() or reduce(): context.getconfiguration().getint("filter", -1); See Hadoop API for other details: Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 24

25 Solution: Mapper Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 25

26 Solution: Reducer Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 26

27 Solution: The Job Setup Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 27

28 References [1] [2] [3] ProgrammingWithHadoop.pdf [4] Happy Coding Oct 19th, 2012 Martin Kaufmann Systems Group, ETH Zürich 28