Hadoop and Big Data. Keijo Heljanko. Department of Information and Computer Science School of Science Aalto University

Transcription

1 Keijo Heljanko Department of Information and Computer Science School of Science Aalto University 1/77

2 Business Drivers of Cloud Computing Large data centers allow for economics of scale Cheaper hardware purchases Cheaper cooling of hardware Example: Google paid 40 MEur for a Summa paper mill site in Hamina, Finland: Data center cooled with sea water from the Baltic Sea Cheaper electricity Cheaper network capacity Smaller number of administrators / computer Unreliable commodity hardware is used Reliability obtained by replication of hardware components and a combined with a fault tolerant software stack 2/77

3 Cloud Computing Technologies A collection of technologies aimed to provide elastic pay as you go computing Virtualization of computing resources: Amazon EC2, Eucalyptus, OpenNebula, Open Stack Compute,... Scalable file storage: Amazon S3, GFS, HDFS,... Scalable batch processing: Google MapReduce, Apache Hadoop, PACT, Microsoft Dryad, Google Pregel, Berkeley Spark,... Scalable datastore:amazon Dynamo, Apache Cassandra, Google Bigtable, HBase,... Distributed Consensus: Google Chubby, Apache Zookeeper,... Scalable Web applications hosting: Google App Engine, Microsoft Azure, Heroku,... 3/77

4 Clock Speed of Processors Herb Sutter: The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software. Dr. Dobb s Journal, 30(3), March 2005 (updated graph in August 2009). 4/77

5 Implications of the End of Free Lunch The clock speeds of microprocessors are not going to improve much in the foreseeable future The efficiency gains in single threaded performance are going to be only moderate The number of transistors in a microprocessor is still growing at a high rate One of the main uses of transistors has been to increase the number of computing cores the processor has The number of cores in a low end workstation (as those employed in large scale datacenters) is going to keep on steadily growing Programming models need to change to efficiently exploit all the available concurrency - scalability to high number of cores/processors will need to be a major focus 5/77

6 Warehouse-scale Computing (WSC) The smallest unit of computation in Google scale is: Warehouse full of computers [WSC]: Luiz André Barroso, Urs Hölzle: The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines Morgan & Claypool Publishers The WSC book says:... we must treat the datacenter itself as one massive warehouse-scale computer (WSC). 6/77

7 Jeffrey Dean (Google): LADIS 2009 keynote failure numbers LADIS 2009 keynote: Designs, Lessons and Advice from Building Large Distributed Systems ladis2009/talks/dean-keynote-ladis2009.pdf At warehouse-scale, failures will be the norm and thus fault tolerant software will be inevitable Hypothetically assume that server mean time between failures (MTBF) would be 30 years (highly optimistic and not realistic number!). In a WSC with nodes roughly one node will fail per day. Typical yearly flakiness metrics from J. Dean (Google, 2009): 1-5% of your disk drives will die Servers will crash at least twice (2-4% failure rate) 7/77

8 Big Data As of May 2009, the amount of digital content in the world is estimated to be 500 Exabytes (500 million TB) EMC sponsored study by IDC in 2007 estimates the amount of information created in 2010 to be 988 EB Worldwide estimated hard disk sales in 2010: 675 million units Data comes from: Video, digital images, sensor data, biological data, Internet sites, social media,... The problem of such large data massed, termed Big Data calls for new approaches to storage and processing of data 8/77

9 Example: Simple Word Search Example: Suppose you need to search for a word in a 2TB worth of text that is found only once in the text mass using a compute cluster Assuming 100MB/s read speed, in the worst case reading all data from a single 2TB disk takes 5.5 hours If 100 hard disks can be used in parallel, the same task takes less than four minutes Scaling using hard disk parallelism is one of the design goals of scalable batch processing in the cloud 9/77

10 Google MapReduce A scalable batch processing framework developed at Google for computing the Web index When dealing with Big Data (a substantial portion of the Internet in the case of Google!), the only viable option is to use hard disks in parallel to store and process it Some of the challenges for storage is coming from Web services to store and distribute pictures and videos We need a system that can effectively utilize hard disk parallelism and hide hard disk and other component failures from the programmer 10/77

11 Google MapReduce (cnt.) MapReduce is tailored for batch processing with hundreds to thousands of machines running in parallel, typical job runtimes are from minutes to hours As an added bonus we would like to get increased programmer productivity compared to each programmer developing their own tools for utilizing hard disk parallelism 11/77

12 Google MapReduce (cnt.) The MapReduce framework takes care of all issues related to parallelization, synchronization, load balancing, and fault tolerance. All these details are hidden from the programmer The system needs to be linearly scalable to thousands of nodes working in parallel. The only way to do this is to have a very restricted programming model where the communication between nodes happens in a carefully controlled fashion Apache Hadoop is an open source MapReduce implementation used by Yahoo!, Facebook, and Twitter 12/77

13 MapReduce and Functional Programming Based on the functional programming in the large: User is only allowed to write side-effect free functions Map and Reduce Re-execution is used for fault tolerance. If a node executing a Map or a Reduce task fails to produce a result due to hardware failure, the task will be re-executed on another node Side effects in functions would make this impossible, as one could not re-create the environment in which the original task executed One just needs a fault tolerant storage of task inputs The functions themselves are usually written in a standard imperative programming language, usually Java 13/77

14 Why No Side-Effects? Side-effect free programs will produce the same output irregardless of the number of computing nodes used by MapReduce Running the code on one machine for debugging purposes produces the same results as running the same code in parallel It is easy to introduce side-effect to MapReduce programs as the framework does not enforce a strict programming methodology. However, the behavior of such programs is undefined by the framework, and should therefore be avoided. 14/77

15 Yahoo! MapReduce Tutorial We use a Figures from the excellent MapReduce tutorial of Yahoo! [YDN-MR], available from: module4.html In functional programming, two list processing concepts are used Mapping a list with a function Reducing a list with a function 15/77

16 Mapping a List Mapping a list applies the mapping function to each list element (in parallel) and outputs the list of mapped list elements: Figure: Mapping a List with a Map Function, Figure 4.1 from [YDN-MR] 16/77

17 Reducing a List Reducing a list iterates over a list sequentially and produces an output created by the reduce function: Figure: Reducing a List with a Reduce Function, Figure 4.2 from [YDN-MR] 17/77

18 Grouping Map Output by Key to Reducer In MapReduce the map function outputs (key, value)-pairs. The MapReduce framework groups map outputs by key, and gives each reduce function instance (key, (..., list of values,...)) pair as input. Note: Each list of values having the same key will be independently processed: Figure: Keys Divide Map Output to Reducers, Figure 4.3 from [YDN-MR] 18/77

19 MapReduce Data Flow Practical MapReduce systems split input data into large (64MB+) blocks fed to user defined map functions: Figure: High Level MapReduce Dataflow, Figure 4.4 from [YDN-MR] 19/77

20 Recap: Map and Reduce Functions The framework only allows a user to write two functions: a Map function and a Reduce function The Map-function is fed blocks of data (block size MB), and it produces (key, value) -pairs The framework groups all values with the same key to a (key, (..., list of values,...)) format, and these are then fed to the Reduce function A special Master node takes care of the scheduling and fault tolerance by re-executing Mappers or Reducers 20/77

21 MapReduce Diagram Worker (1)fork (2)assign map User program (1)fork Master (2) assign reduce (1)fork split 0 split 1 (3)read split 2 Worker (4) local write (5)Remote read Worker (6)write output file 0 split 3 split 4 Worker output file 1 Worker Input files Map phase Intermediate files (on local disks) Reduce phase Output files Figure: J. Dean and S. Ghemawat: MapReduce: Simplified Data Processing on Large Clusters, OSDI /77

22 Example: Word Count Classic word count example from the Hadoop MapReduce tutorial: current/mapred_tutorial.html Consider doing a word count of the following file using MapReduce: Hello World Bye World Hello Hadoop Goodbye Hadoop 22/77

23 Example: Word Count (cnt.) Consider a Map function that reads in words one a time, and outputs (word, 1) for each parsed input word The Map function output is: (Hello, 1) (World, 1) (Bye, 1) (World, 1) (Hello, 1) (Hadoop, 1) (Goodbye, 1) (Hadoop, 1) 23/77

24 Example: Word Count (cnt.) The Shuffle phase between Map and Reduce phase creates a list of values associated with each key The Reduce function input is: (Bye, (1)) (Goodbye, (1)) (Hadoop, (1, 1)) (Hello, (1, 1)) (World, (1, 1)) 24/77

25 Example: Word Count (cnt.) Consider a reduce function that sums the numbers in the list for each key and outputs (word, count) pairs. The output of the Reduce function is the output of the MapReduce job: (Bye, 1) (Goodbye, 1) (Hadoop, 2) (Hello, 2) (World, 2) 25/77

26 Phases of MapReduce 1. A Master (In Hadoop terminology: Job Tracker) is started that coordinates the execution of a MapReduce job. Note: Master is a single point of failure 2. The master creates a predefined number of M Map workers, and assigns each one an input split to work on. It also later starts a predefined number of R reduce workers 3. Input is assigned to a free Map worker MB split at a time, and each user defined Map function is fed (key, value) pairs as input and also produces (key, value) pairs 26/77

27 Phases of MapReduce(cnt.) 4. Periodically the Map workers flush their (key, value) pairs to the local hard disks, partitioning by their key to R partitions (default: use hashing), one per reduce worker 5. When all the input splits have been processed, a Shuffle phase starts where M R file transfers are used to send all of the mapper outputs to the reducer handling each key partition. After reducer receives the input files, reducer sorts (and groups) the (key, value) pairs by the key 6. User defined Reduce functions iterate over the (key, (..., list of values,...)) lists, generating output (key, value) pairs files, one per reducer 27/77

28 Google MapReduce (cnt.) The user just supplies the Map and Reduce functions, nothing more The only means of communication between nodes is through the shuffle from a mapper to a reducer The framework can be used to implement a distributed sorting algorithm by using a custom partitioning function The framework does automatic parallelization and fault tolerance by using a centralized Job tracker (Master) and a distributed filesystem that stores all data redundantly on compute nodes Uses functional programming paradigm to guarantee correctness of parallelization and to implement fault-tolerance by re-execution 28/77

29 Apache Hadoop An Open Source implementation of the MapReduce framework, originally developed by Doug Cutting and heavily used by e.g., Yahoo! and Facebook Moving Computation is Cheaper than Moving Data - Ship code to data, not data to code. Map and Reduce workers are also storage nodes for the underlying distributed filesystem: Job allocation is first tried to a node having a copy of the data, and if that fails, then to a node in the same rack (to maximize network bandwidth) Project Web page: 29/77

30 Apache Hadoop (cnt.) When deciding whether MapReduce is the correct fit for an algorithm, one has to remember the fixed data-flow pattern of MapReduce. The algorithm has to be efficiently mapped to this data-flow pattern in order to efficiently use the underlying computing hardware Builds reliable systems out of unreliable commodity hardware by replicating most components (exceptions: Master/Job Tracker and NameNode in Hadoop Distributed File System) 30/77

31 Apache Hadoop (cnt.) Tuned for large (gigabytes of data) files Designed for very large 1 PB+ data sets Designed for streaming data accesses in batch processing, designed for high bandwidth instead of low latency For scalability: NOT a POSIX filesystem Written in Java, runs as a set of user-space daemons 31/77

32 Hadoop Distributed Filesystem (HDFS) A distributed replicated filesystem: All data is replicated by default on three different Data Nodes Inspired by the Google Filesystem Each node is usually a Linux compute node with a small number of hard disks (4-12) A single NameNode that maintains the file locations, many DataNodes (1000+) 32/77

33 Hadoop Distributed Filesystem (cnt.) Any piece of data is available if at least one datanode replica is up and running Rack optimized: by default one replica written locally, second in the same rack, and a third replica in another rack (to combat against rack failures, e.g., rack switch or rack power feed) Uses large block size, 128 MB is a common default - designed for batch processing For scalability: Write once, read many filesystem 33/77

34 Implications of Write Once All applications need to be re-engineered to only do sequential writes. Example systems working on top of HDFS: HBase (Hadoop Database), a database system with only sequential writes, Google Bigtable clone MapReduce batch processing system Apache Pig and Hive data mining tools Mahout machine learning libraries Lucene and Solr full text search Nutch web crawling 34/77

35 Two Large Hadoop Installations Yahoo! (2009): 4000 nodes, 16 PB raw disk, 64TB RAM, 32K cores Facebook (2010): 2000 nodes, 21 PB storage, 64TB RAM, 22.4K cores 12 TB (compressed) data added per day, 800TB (compressed) data scanned per day A. Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. Sen Sarma, R. Murthy, H. Liu: Data warehousing and analytics infrastructure at Facebook. SIGMOD Conference 2010: /77

36 Writing Hadoop Jobs See Hadoop installation instructions at: /practical_matters Example: Assume we want to count the words in a file using Apache Hadoop What we need to write in Java: Code to include the needed Hadoop Java Libraries Code for the Mapper Code for the Reducer (Optionally: Code for the Combiner) Code for the main method for Job configuration and submission 36/77

37 WordCount: Including Hadoop Java Libraries package org.apache.hadoop.examples; import java.io.ioexception; import java.util.stringtokenizer; import org.apache.hadoop.conf.configuration; import org.apache.hadoop.fs.path; import org.apache.hadoop.io.intwritable; import org.apache.hadoop.io.text; import org.apache.hadoop.mapreduce.job; import org.apache.hadoop.mapreduce.mapper; import org.apache.hadoop.mapreduce.reducer; import org.apache.hadoop.mapreduce.lib.input. FileInputFormat; import org.apache.hadoop.mapreduce.lib.output. FileOutputFormat; import org.apache.hadoop.util.genericoptionsparser; 37/77

38 WordCount: Including Hadoop Java Libraries The package org.apache.hadoop.examples gives the package name we are defining The java.util.stringtokenizer is a utility to parse a string into a list of words by returning an iterator over the words of the string Most org.apache.hadoop.* libraries listed are needed by all Hadoop jobs, cut&paste from examples 38/77

39 WordCount: Including Hadoop Java Libraries The IntWritable is a Hadoop variant of the Java Integer type. The Text is a Hadoop variant of the Java String class In addition: the LongWritable is a Hadoop variant of the Java Long type, the FloatWritable is a Hadoop variant of the Java Float type, etc. The Hadoop types must be used for both Mapper and Reducer keys and values instead of Java native types 39/77

40 WordCount: Code for the Mapper public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); } public void map(object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasmoretokens()) { word.set(itr.nexttoken()); context.write(word, one); } } 40/77

41 WordCount: Code for the Mapper The line public class WordCount tells we are defining the WordCount class, into which Mapper, Reducer, and main method all belong into The line public static class TokenizerMapper tells the name of the Mapper we are defining The four parameters of the Mapper in: extends Mapper<Object, Text, Text, IntWritable> are the types of the input key, input value, output key, and output value, respectively The key argument is not used by the WordCount Mapper, but by default it is a long integer offset for text inputs 41/77

42 WordCount: Code for the Mapper Remember that Hadoop keys and values are of Hadoop internal types: Text, IntWritable, LongWritable, FloatWritable, etc. Code is needed to covert between them and Java native types. Because of this, define the Hadoop types for creating the Mapper output: The code: private final static IntWritable one = new IntWritable(1) creates an IntWritable constant 1 The code: private Text word = new Text() creates a Text object to be used as output key object of the Mapper 42/77

43 WordCount: Code for the Mapper The Mapper method is defined as: public void map( Object key, Text value, Context context), where Context is an object used to write the output of the Mapper The Java code: StringTokenizer itr = new StringTokenizer(value.toString()) parses a Text value converted into Java String into a list of words by returning an iterator itr over the words of the String The code word.set(itr.nexttoken()) overwrites the current contents of the Text object word by the next String contents returned from the iterator itr The code context.write(word, one) outputs the parsed word as key, and the constant 1 as the value from the Mapper giving an output pair (word,1) of (Text, IntWritable) type 43/77

44 WordCount: Code for the Reducer public static class IntSumReducer extends Reducer<Text,IntWritable,Text, IntWritable> { private IntWritable result = new IntWritable(); } public void reduce(text key, Iterable<IntWritable > values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } 44/77

45 WordCount: Code for the Reducer The line public static class IntSumReducer gives the name of the Reducer The four parameters of the Reducer in: extends Reducer<Text,IntWritable,Text,IntWritable> are the types of the input key, input value, output key, and output value, respectively. The input types of Reducer must match the output types of the Mapper The code private IntWritable result = new IntWritable() generates a IntWritable object to be used as output value of the Reducer 45/77

46 WordCount: Code for the Reducer The Reducer method is defined as: public void reduce(text key, Iterable<IntWritable> values, Context context), where Iterable<IntWritable> values is an iterator over the value type of the Mapper output, and Context is an object used to write the output of the Reducer The reducer will be called once per each key, and will be given a list of values mapped to that key to iterate on using the values iterator 46/77

47 WordCount: Code for the Reducer The loop for (IntWritable val : values) iterates over the values mapped to the current key and sums them up The code result.set(sum) basically sets the IntWritable object result using the sum variable, remember that native Java classes can not be directly used as outputs The code context.write(key, result) writes the Reducer output as a (word, count) pair of (Text, IntWritable) type 47/77

48 WordCount: Code for main 1/2 public static void main(string[] args) throws Exception { Configuration conf = new Configuration(); String[] otherargs = new GenericOptionsParser( conf, args).getremainingargs(); if (otherargs.length!= 2) { System.err.println("Usage: wordcount <in> <out >"); System.exit(2); } 48/77

49 WordCount: Code for main 1/2 The line public static void main(string[] args) declares the main routine that will be running on the client machine that starts the MapReduce job The line Configuration conf = new Configuration() creates the configuration object conf holding all the information of the MapReduce job to be created The line String[] otherargs = new GenericOptionsParser (conf, args).getremainingargs() and code after it does argument processing and checking that the code is called with two arguments 49/77

50 WordCount: Code for main 2/2 } } Job job = new Job(conf, "word count"); job.setjarbyclass(wordcount.class); job.setmapperclass(tokenizermapper.class); job.setcombinerclass(intsumreducer.class); job.setreducerclass(intsumreducer.class); job.setoutputkeyclass(text.class); job.setoutputvalueclass(intwritable.class); FileInputFormat.addInputPath(job, new Path( otherargs[0])); FileOutputFormat.setOutputPath(job, new Path( otherargs[1])); System.exit(job.waitForCompletion(true)? 0 : 1); 50/77

51 WordCount: Code for main 2/2 The code Job job = new Job(conf, "word count") creates a new MapReduce Job with name word count using the created configuration object conf The line job.setjarbyclass(wordcount.class) tells the name of the defined class The Mapper is set: job.setmapperclass(tokenizermapper.class) The (optional!) Combiner is set: job.setcombinerclass(intsumreducer.class), note that this is sound only when the Reduce function is commutative and associative! (Addition is!) The Reducer is set: job.setreducerclass(intsumreducer.class) 51/77

52 WordCount: Code for main 2/2 Java type system is not able to figure out correct types of output keys and values, so we have to specify them again: The code job.setoutputkeyclass(text.class) sets the output key of both Mapper and Reducer to the Text type. If Mapper has a different output key type from Reducer, the Mapper key type can be specified by overriding this using job.setmapoutputkeyclass() The code job.setoutputvalueclass(intwritable.class) sets the output key of both Mapper and Reducer to the IntWritable type. Also job.setmapoutputvalueclass() can be used to override this if the Mapper output type is different Also because of the Java limitations, it does not complain at compile time if Mapper outputs and Reducer inputs are of incompatible types 52/77

53 WordCount: Code for main 2/2 The code FileInputFormat.addInputPath(job, new Path( otherargs[0])) sets the input to be all of files in input directory specified as first argument on command line The code FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])) sets the output directed to the output directory specified as second argument on command line, one file per reducer The code System.exit(job.waitForCompletion(true)? 0 : 1) submits the job to Hadoop, waits for the job to finish, and returns 0 on success, 1 on failure 53/77

54 A Quick WordCount Demo: Formatting HDFS $ hadoop namenode -format 11/09/20 01:39:29 INFO namenode.namenode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = nipsu.local/ STARTUP_MSG: args = [-format] STARTUP_MSG: version = STARTUP_MSG: build = branch-0.20-security-203 -r ; compiled by oom on Wed May 4 07:57:50 PDT 2011 ************************************************************/ 11/09/20 01:39:29 INFO util.gset: VM type = 64-bit 11/09/20 01:39:29 INFO util.gset: 2% max memory = MB 11/09/20 01:39:29 INFO util.gset: capacity = 2ˆ21 = entries 11/09/20 01:39:29 INFO util.gset: recommended= , actual= /09/20 01:39:30 INFO namenode.fsnamesystem: fsowner=keijoheljanko 11/09/20 01:39:30 INFO namenode.fsnamesystem: supergroup=supergroup 11/09/20 01:39:30 INFO namenode.fsnamesystem: ispermissionenabled=true 11/09/20 01:39:30 INFO namenode.fsnamesystem: dfs.block.invalidate.limit=100 11/09/20 01:39:30 INFO namenode.fsnamesystem: isaccesstokenenabled=false accesskeyupdateinterval=0 min(s), accesstokenlifetime=0 min(s) 11/09/20 01:39:30 INFO namenode.namenode: Caching file names occuring more than 10 times 11/09/20 01:39:30 INFO common.storage: Image file of size 119 saved in 0 seconds. 11/09/20 01:39:30 INFO common.storage: Storage directory /tmp/hadoop-keijoheljanko/dfs/name has been successfully formatted. 11/09/20 01:39:30 INFO namenode.namenode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at nipsu.local/ ************************************************************/ 54/77

55 A Quick WordCount Demo: Starting Hadoop $ start-all.sh starting namenode, logging to /Users/keijoheljanko/hadoop/hadoop /bin/.. /logs/hadoop-keijoheljanko-namenode-nipsu.local.out localhost: starting datanode, logging to /Users/keijoheljanko/hadoop/hadoop /bin/.. /logs/hadoop-keijoheljanko-datanode-nipsu.local.out localhost: starting secondarynamenode, logging to /Users/keijoheljanko/hadoop/hadoop /bin/.. /logs/hadoop-keijoheljanko-secondarynamenode-nipsu.local.out starting jobtracker, logging to /Users/keijoheljanko/hadoop/hadoop /bin/.. /logs/hadoop-keijoheljanko-jobtracker-nipsu.local.out localhost: starting tasktracker, logging to /Users/keijoheljanko/hadoop/hadoop /bin/.. /logs/hadoop-keijoheljanko-tasktracker-nipsu.local.out 55/77

56 A Quick WordCount Demo: Uploading Kalevala to HDFS $ wget :41: Resolving ( , 2001:708:10:9::20:3 Connecting to ( :80... connected. HTTP request sent, awaiting response OK Length: (196K) [application/x-gzip] Saving to: kalevala.txt.gz 100%[===============================================================================>] 200, :41:24 (3.06 MB/s) - kalevala.txt.gz saved [200872/200872] $ gunzip kalevala.txt.gz $ recode ISO UTF-8 kalevala.txt $ ls -al kalevala.txt -rw-r--r-- 1 keijoheljanko staff Nov kalevala.txt $ hadoop fs -mkdir input $ hadoop fs -copyfromlocal kalevala.txt input $ hadoop fs -ls input Found 1 items -rw-r--r-- 1 keijoheljanko supergroup :46 /user/keijoheljanko/input/ 56/77

57 A Quick WordCount Demo: Compiling WordCount $ ls -al WordCount.java -rw-r--r--@ 1 keijoheljanko staff 2395 Sep 18 13:53 WordCount.java $ javac -classpath ${HADOOP_HOME}/hadoop-core-${HADOOP_VERSION}.jar:{HADOOP_HOME}/lib/commons-cli-1.2.jar -d wordcount_classes WordCount.java $ jar -cvf wordcount.jar -C wordcount_classes/. added manifest adding: org/(in = 0) (out= 0)(stored 0%) adding: org/apache/(in = 0) (out= 0)(stored 0%) adding: org/apache/hadoop/(in = 0) (out= 0)(stored 0%) adding: org/apache/hadoop/examples/(in = 0) (out= 0)(stored 0%) adding: org/apache/hadoop/examples/wordcount$intsumreducer.class(in = 1789) (out= 746)(deflated 58%) adding: org/apache/hadoop/examples/wordcount$tokenizermapper.class(in = 1790) (out= 765)(deflated 57%) adding: org/apache/hadoop/examples/wordcount.class(in = 1911) (out= 996)(deflated 47%) $ ls -al wordcount.jar -rw-r--r-- 1 keijoheljanko staff 3853 Sep 20 02:07 wordcount.jar 57/77

58 A Quick WordCount Demo: Running WordCount (1/2) $ hadoop jar wordcount.jar org.apache.hadoop.examples.wordcount input output 11/09/20 02:11:32 INFO input.fileinputformat: Total input paths to process : 1 11/09/20 02:11:32 INFO mapred.jobclient: Running job: job_ _ /09/20 02:11:33 INFO mapred.jobclient: map 0% reduce 0% 11/09/20 02:11:47 INFO mapred.jobclient: map 100% reduce 0% 11/09/20 02:12:00 INFO mapred.jobclient: map 100% reduce 100% 11/09/20 02:12:05 INFO mapred.jobclient: Job complete: job_ _ /09/20 02:12:05 INFO mapred.jobclient: Counters: 25 11/09/20 02:12:05 INFO mapred.jobclient: Job Counters 11/09/20 02:12:05 INFO mapred.jobclient: Launched reduce tasks=1 11/09/20 02:12:05 INFO mapred.jobclient: SLOTS_MILLIS_MAPS= /09/20 02:12:05 INFO mapred.jobclient: Total time spent by all reduces waiting after reserving slots (ms)=0 11/09/20 02:12:05 INFO mapred.jobclient: Total time spent by all maps waiting after reserving slots (ms)=0 11/09/20 02:12:05 INFO mapred.jobclient: Launched map tasks=1 11/09/20 02:12:05 INFO mapred.jobclient: Data-local map tasks=1 11/09/20 02:12:05 INFO mapred.jobclient: SLOTS_MILLIS_REDUCES= /09/20 02:12:05 INFO mapred.jobclient: File Output Format Counters 11/09/20 02:12:05 INFO mapred.jobclient: Bytes Written= /77

59 A Quick WordCount Demo: Running WordCount (2/2) 11/09/20 02:12:05 INFO mapred.jobclient: FileSystemCounters 11/09/20 02:12:05 INFO mapred.jobclient: FILE_BYTES_READ= /09/20 02:12:05 INFO mapred.jobclient: HDFS_BYTES_READ= /09/20 02:12:05 INFO mapred.jobclient: FILE_BYTES_WRITTEN= /09/20 02:12:05 INFO mapred.jobclient: HDFS_BYTES_WRITTEN= /09/20 02:12:05 INFO mapred.jobclient: File Input Format Counters 11/09/20 02:12:05 INFO mapred.jobclient: Bytes Read= /09/20 02:12:05 INFO mapred.jobclient: Map-Reduce Framework 11/09/20 02:12:05 INFO mapred.jobclient: Reduce input groups= /09/20 02:12:05 INFO mapred.jobclient: Map output materialized bytes= /09/20 02:12:05 INFO mapred.jobclient: Combine output records= /09/20 02:12:05 INFO mapred.jobclient: Map input records= /09/20 02:12:05 INFO mapred.jobclient: Reduce shuffle bytes= /09/20 02:12:05 INFO mapred.jobclient: Reduce output records= /09/20 02:12:05 INFO mapred.jobclient: Spilled Records= /09/20 02:12:05 INFO mapred.jobclient: Map output bytes= /09/20 02:12:05 INFO mapred.jobclient: Combine input records= /09/20 02:12:05 INFO mapred.jobclient: Map output records= /09/20 02:12:05 INFO mapred.jobclient: SPLIT_RAW_BYTES=124 11/09/20 02:12:05 INFO mapred.jobclient: Reduce input records= /77

60 A Quick WordCount Demo: Dowloading Output $ hadoop fs -copytolocal output output $ ls -al output total 624 drwxr-xr-x 5 keijoheljanko staff 170 Sep 20 02:16. drwxr-xr-x 17 keijoheljanko staff 578 Sep 20 02:16.. -rw-r--r-- 1 keijoheljanko staff 0 Sep 20 02:16 _SUCCESS drwxr-xr-x 3 keijoheljanko staff 102 Sep 20 02:16 _logs -rw-r--r-- 1 keijoheljanko staff Sep 20 02:16 part-r $ head -15 output/part-r "Ahto,1 "Aik 1 "Aika 1 "Ain 1 "Aina 2 "Ainap 1 "Ainapa 1 "Aita 2 "Ajoa 1 "Akat 1 "Akatp 1 "Akka 5 "Ala 1 "Ampuisitko 1 "Anna 7 60/77

61 Hadoop Books I can warmly recommend the Book: Tom White: Hadoop: The Definitive Guide, Second Edition, O Reilly Media, Print ISBN: , Ebook ISBN: An alternative book is: Chuck Lam: Hadoop in Action, Manning, Print ISBN: /77

62 Recap of Hadoop Commands Used start-all.sh Starts all Hadoop daemons. Hint: If you are getting the following error on Mac OSX: Bad connection to FS. command aborted. exception: Call to localhost/ :9000 failed on connection exception: java.net.connectexception: Connection refused call start-all.sh another time to start the namenode for real. 62/77

63 Recap of Hadoop Commands Used (cnt.) hadoop namenode -format Formats the HDFS filesystem hadoop fs -mkdir foo Creates a directory called foo in the HDFS home directory of the user hadoop fs -rmr foo Removes the directory/file called foo in the HDFS home directory of the user hadoop fs -copyfromlocal foo.txt bar Copies the file foo.txt to the HDFS directory/file called bar in the HDFS home directory of the user 63/77

64 Recap of Hadoop Commands Used (cnt.) hadoop fs -ls foo Does a directory/file listing for directory called foo in the HDFS home directory of the user hadoop jar wordcount.jar org.apache.hadoop.examples.wordcount foo bar Runs MapReduce job in file wordcount.jar in the class org.apache.hadoop.examples.wordcount using the HDFS directory foo as input and stores the output in a new HDFS directory bar to be created by the job (i.e., bar should not yet exist in HDFS) 64/77

65 Recap of Hadoop Commands Used (cnt.) hadoop fs -copytolocal foo bar Copies the directory/file foo in the user HDFS home directory to the local file system directory/file bar stop-all.sh Stops all Hadoop daemons 65/77

66 Hadoop Default ports Use the URL: to access the JobTracker through a Web browser. Hint: Use the refresh button to update the view during Job execution. Use the URL: to access the NameNode through a Web browser. Hint: You can also browse HDFS filesystem through this URL. 66/77

67 Hadoop NameNode and SecondaryNameNode Locking Hadoop Namenode and SecondaryNameNode use locking to disallow the accidental starting of two daemons modifying the same files and leading to HDFS inconsistency If either daemon gets killed uncleanly, it they might leave lock files around and prevent a new NameNode or SecondaryNameNode from starting The lock files are called data/in use.lock, name/in use.lock, and namesecondary/in use.lock. They are located in dfs subdirectory of hadoop.tmp.dir that can be set in core-site.xml (by default /tmp/hadoop-${user.name}) 67/77

68 Commercial Hadoop Support Cloudera: A Hadoop distribution based on Apache Hadoop + patches. Available from: Hortonworks: Yahoo! spin-off of their Hadoop development team, will be working on mainline Apache Hadoop. MapR: A rewrite of much of Apache Hadoop in C++, including a new filesystem. API-compatible with Apache Hadoop. 68/77

69 Cloud Software Project ICT SHOK Program Cloud Software ( ) A large Finnish consortium, see: Case study at Aalto: CSC Genome Browser Cloud Backend Co-authors: Matti Niemenmaa, André Schumacher (Aalto University, Department of Information and Computer Science), Aleksi Kallio, Eija Korpelainen, Taavi Hupponen, and Petri Klemelä (CSC IT Center for Science) 69/77

70 CSC Genome Browser CSC provides tools and infrastructure for bioinformatics Bioinformatics is the largest customer group of CSC (in user numbers) Next-Generation Sequencing (NGS) produces large data sets (TB+) Cloud computing can be harnessed for analyzing these data sets 1000 Genomes project ( Freely available 50 TB+ data set of human genomes 70/77

71 CSC Genome Browser Cloud computing technologies will enable scalable NGS data analysis There exists prior research into sequence alignment and assembly in the cloud Visualization is needed in order to understand large data masses Interactive visualization for 100GB+ datasets can only be achieved with preprocessing in the cloud 71/77

72 Genome Browser Requirements Interactive browsing with zooming in and out, Google Earth -style Single datasets 100GB-1TB+ with interactive visualization at different zoom levels Preprocessing used to compute summary data for the higher zoom levels Dataset too large to compute the summary data in real time using the real dataset Scalable cloud data processing paradigm map-reduce implemented in Hadoop used to compute the summary data in preprocessing (currently upto 15 x 12 = 180 cores) 72/77

73 Genome Browser GUI 73/77

74 Experiments One 50 GB (compressed) BAM input file from 1000 Genomes Run on the Triton cluster 1-15 compute nodes with 12 cores each Four repetitions for increasing number of worker nodes Two types of runs: sorting according to starting position ( sorted ) and read aggregation using five summary-block sizes at once ( summarized ) 74/77

75 Mean speedup 50 GB sorted 50 GB summarized for B=2,4,8,16, Ideal Input file import Sorting Output file export Total elapsed Ideal Input file import Summarizing Output file export Total elapsed Mean speedup Mean speedup Workers Workers 75/77

76 Results An updated CSC Genome browser GUI Chipster: Tools and framework for producing visualizable data An implementation of preprocessing for visualization using Hadoop Scalability studies for running Hadoop on 50 GB+ datasets on the Triton cloud testbed Software released, with more than 1200 downloads: 76/77

77 Current Research Topics High level programming frameworks for ad-hoc queries: Apache Pig integration of Hadoop-BAM in the SeqPig tool ( and SQL integration in Apache Hive Real-time SQL query evaluation on Big Data: Cloudera Impala and Berkeley Shark Scalable datastores: HBase (Hadoop Database) for real-time read/write database workloads Energy efficiency through better main memory and SSD utilization 77/77