Fewer Hoops with Hadoop: Spring & Hadoop

Transcription

1 Fewer Hoops with Hadoop: Spring & Hadoop Costin Leau, Staff Engineer 2009 VMware Inc. All rights reserved

2 Goal of the talk Improve your Hadoop experience through a POJO-based programming model 2

3 Goal of the talk Improve your Hadoop experience through a POJO-based programming model Hear your experiences with Hadoop 3

4 Agenda Using Hadoop POJO programming Integration points 4

5 About Costin Spring committer since 2006 Involved in: Spring Framework Pitchfork (Spring-based EJB 3 support in WebLogic) Spring JavaConfig Spring OSGi Spring Data Spring GemFire 5

6 Using Hadoop 6

7 What is Hadoop? Framework for writing applications which process vast amounts of data in-parallel Exponential growth of data ZettaByte (10^21 or 2^80) reached in 2010 Forecast for ZB traffic per year for 2015 Inspired by Google File System (GFS) and Map Reduce papers Umbrella for several sub-projects: Hadoop Common HDFS MapReduce Related projects: Avro, Hbase, Hive, ZooKeeper 7

8 It s all about Map/Reduce Simple abstract for large-scale computation * image taken from Hadoop definitive guide by Tom White 8

9 Counting words (the MR way) public class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasmoretokens()) { word.set(itr.nexttoken());context.write(word, one); }} } public class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } 9

10 Too much noise Leaking abstraction Need to extend/implement existing interface Restricted Argument Types Usage of custom classes Due to serialization Conversion required 10

11 Spring (&) Hadoop 11

12 Spring Framework project at a glance Provides: IoC container - configuration, wiring and lifecycle management of Java objs AOP applying cross-cutting routines (dynamic decoration) Service abstractions TX mgt., MVC fwk, Remoting, Data Access Promotes POJO programming model Eliminates infrastructure/configuration code Increases portability and component reusability Example: a class can be made transaction w/o knowing the TX API Reduces boiler-plate code Increases productivity Open Source (Apache License 2.0) Not just Java ported to.net, Python, ActionScript 12

13 Unifying themes 13

14 Spring Ecosystem 14

15 Spring Ecosystem 15

16 POJO FTW! public class PojoMapReducer public void map(string value, Map<String, Integer> writer) { StringTokenizer itr = new StringTokenizer(value); while (itr.hasmoretokens()) { writer.put(itr.nexttoken(), 1);} } public int reduce(iterable<integer> values) { int sum = 0; for (Integer val : values) { sum += val; } return sum; } 16

17 Spring Hadoop project Promote classic Spring value propositions Productivity remove API noise, resource management Make the easy stuff a one-liner while allow access to lower level APIs Consistency/ Portability / Support POJO style programming model for Hadoop MapReduce jobs No need to implement an interface or extend an abstract class (favors composition vs inheritance) No need to use Hadoop dedicated *Writeable types (just use the JDK types) Dependency injection for MapReduce Jobs Simplifies considerably configuration and provisioning of non-trivial jobs Integration with HDFS Navigate and extract data from the underlying Hadoop format Integration with Spring Batch and Integration 17

18 Open and Free (no strings attached) Open Source Friendly License Apache License 2.0 Easy to embed and extend 18

19 Simplifying MR Remove the need to implement or extend an interface Still an option if you like it Automatic conversion of known types to Hadoop *writable JDK classes Easily pluggable Works great in Web Apps (since they are likely to use custom converters) Infers type arguments based an your method You decide the param order and type The framework take care of the conversion for you 19

20 POJO with internal configuration public class PojoMapReducer public void map(string value, Map<String, Integer> writer) { StringTokenizer itr = new StringTokenizer(value); while (itr.hasmoretokens()) { writer.put(itr.nexttoken(), 1);} } public int reduce(iterable<integer> values) { int sum = 0; for (Integer val : values) { sum += val; } return sum; } <context:component-scan base-package= com.company.hadoop" /> 20

21 POJO with external configuration public class PojoMapReducer { public void map(string value, Map<String, Integer> writer) { StringTokenizer itr = new StringTokenizer(value); while (itr.hasmoretokens()) { writer.put(itr.nexttoken(), 1);} } } public int reduce(iterable<integer> values) { int sum = 0; for (Integer val : values) { sum += val; } return sum; } <hadoop:mapper id="mapper ref="pojomapper" method="map" /> <hadoop:reducer id= reducer ref="pojoreducer" method= reduce" /> 21

22 Ease of configuring jobs A Hadoop job has a lot of parameters Most of them though are the same across jobs Great candidate for IoC Auto-wiring of properties (based on existing & default definitions) No need to specify combiners, grouping, Input/OutputFormat, etc Externalized environment specific items Easy templating <hadoop:job mapper-ref="mapper reducer-ref="reducer input-paths= ${hadoop.in} output-path= ${hadoop.out} /> <context:property-placeholder properties-ref="bootstrap" /> 22

23 Using Hadoop components Easily plug-in different Hadoop stores HDFS HBase Execute Pig/Hive queries Handle input/output Start/stop/configure query engine instances Locating/configuring the scripts (Python/JavaScript) 23

24 Integration points 24

25 Data Workflow Hadoop jobs usually part of a data workflow Every X days, process the last added data When a file arrives, start the process If a certain result is found, send a notification Jobs need to be created, canceled, retried, stopped 25

26 Spring Batch Problem: Apply well known technologies to old batch problems, and modernize batch architectures. Solution: Extend the Spring programming model to deliver Batch processing in Java Concept Job, Steps ItemReader & ItemWriter Repeat/Skip/ Retry Partitioning Description Declarative Batch programming Reusable building block for common tedious tasks (files, CSV, fixed length, SQL, XML, filtering, sorting ) Batch flow Leverage infrastructure to scale batch processing 26

27 Spring Batch Concepts Infrastructure: low-level concepts Input, Output, Retry, Repeat Core: boundaries and management information Job, Step, Launch Offers sequential, conditional, parallel execution 27

28 Spring Batch 28

29 Spring Integration POJO based model for EIP Provides implementation of classic EIP patterns, e.g. Routing Transformation Innovative annotation model 29

30 Message Endpoint Types Transformer Convert payload or modify headers Filter Discard messages based on boolean evaluation Router Determine next channel based on content Splitter Generate multiple messages from one Aggregator Assemble a single message from multiple 30

31 Spring Batch + Integration Spring Integration Pattern Splitter Aggregator Wire Tap CBR Transformer Description Split one message into multiple messages Combine multiple messages into a single message Inspect point to point messages Route a message based on content Transform message content into a different form Use cases with Spring Batch Message triggers Job Step wraps gateway (e.g. background processing) Remote chunking, Partitioning Automatic retry Flat file and XML processing QoS: once only processing, restart Audit: track status of execution 31

32 Hadoop integration Create Jobs on demand Read/Write job configuration and data from/to HDFS Pig scripts Hive queries Streaming (python/javascript/ruby) Send notifications to/from various gateways/adapters Consume data no matter the transport/source 32

33 Thank Booth #