COSC 6397 Big Data Analytics. 2 nd homework assignment Pig and Hive. Edgar Gabriel Spring 2015

Transcription

1 COSC 6397 Big Data Analytics 2 nd homework assignment Pig and Hive Edgar Gabriel Spring nd Homework Rules Each student should deliver Source code (.java files) Documentation (.pdf,.doc,.tex or.txt file) explanations to the code answers to questions Deliver electronically to [email protected] Expected by Friday, March 31, 11.59pm In case of questions: ask, ask, ask! 1

2 Part 1: Bulk Import into HBase 1. Given the same stack exchange data sets as used during the 1 st homework assignment. Develop a MapReduce code to perform a bulk import of the data in /bigdata-hw2 (not the large file from hw1) Same format as in the 1 st assignment, just different size ( ~220 MB). you have to decide on the schema, e.g. whether and how to group columns in families etc. Justify your selection in the document! Additional information: New build.xml example available on webpage, since you have to utilize different directories for the hbase jar files Additional argument required when starting a job to tell yarn where the hbase jar files are gabriel@shark> HADOOP_CLASSPATH=`hbase classpath` yarn jar edgarairquality.jar airqualitydemo.airquality /gabriel/input /gabriel/output Requirements: Final table name has to have the format bigd<xy>-hbasesample to avoid name clash of student tables 2

3 Required operations need to create first a table manually setting up the schema Perform MapReduce job using HFileOutputformat Input data using bulkload tool Document all steps in details Since not everything that you do is in the java file, please make sure that you document precisely how you created the table, how you executed the bulkload etc. Part 2 Write a MapReduce code, which calculate the average number of answers per question using the data from HBase (instead of the file) Compare the performance of the Hbase version of the code with your version from the 1 st homework assignment for the new file 3

4 Documentation The Documentation should contain (Brief) Problem description Solution strategy Results section Description of resources used Description of measurements performed Results (graphs/tables + findings) The document should not contain Replication of the entire source code that s why you have to deliver the sources Screen shots of every single measurement you made Actually, no screen shots at all. The output files 4

5 Pig Pig is an abstraction on top of Hadoop Pig is a platform for analyzing large data sets Provides high level programming language designed for data processing Converted into MapReduce and executed on Hadoop Clusters Why using Pig? MapReduce requires programmers Must think in terms of map and reduce functions More than likely will require Java programming Pig provides high-level language that can be used by Analysts and Scientists Does not require know how in parallel programming Pig s Features Join Datasets Sort Datasets Filter Data Types Group By User Defined Functions 5

6 Pig Components Pig Latin Command based language Designed specifically for data transformation and flow expression Execution Environment The environment in which Pig Latin commands are executed Supporting local and Hadoop execution modes Pig compiler converts Pig Latin to MapReduce Automatic vs. user level optimizations compared to manual MapReduce code Running Pig Script Execute commands in a file $pig scriptfile.pig Grunt Interactive Shell for executing Pig Commands Started when script file is NOT provided Can execute scripts from Grunt via run or exec commands Embedded Execute Pig commands using PigServer class Can have programmatic access to Grunt via PigRunner class 6

7 Pig Latin concepts Building blocks Field piece of data Tuple ordered set of fields, represented with ( and ) (10.4, 5, word, 4, field1) Bag collection of tuples, represented with { and } { (10.4, 5, word, 4, field1), (this, 1, blah) } Similar to Relational Database Bag is a table in the database Tuple is a row in a table $ pig grunt> cat /input/pig/a.txt a 1 d 4 c 9 k 6 Simple Pig Latin example Load grunt in default map-reduce mode grunt supports file system commands Load contents of text file into a bag called records grunt> records = LOAD '/input/a.txt' as (letter:chararray, count:int); grunt> dump records;... org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - 50% complete :36:22,040 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - 100% complete... (a,1) (d,4) (c,9) (k,6) grunt> Display records on screen 7

8 Simple Pig Latin example No action is taken until DUMP or STORE commands are encountered Pig will parse, validate and analyze statements but not execute them STORE saves results (typically to a file) DUMP displays the results to the screen doesn t make sense to print large arrays to the screen For information and debugging purposes you can print a small sub-set to the screen grunt> records = LOAD '/input/excite-small.log' AS (userid:chararray, timestamp:long, query:chararray); grunt> toprint = LIMIT records 5; grunt> DUMP toprint; Simple Pig Latin example LOAD 'data' [USING function] [AS schema]; data name of the directory or file Must be in single quotes USING specifies the load function to use By default uses PigStorage which parses each line into fields using a delimiter Default delimiter is tab ( \t ) The delimiter can be customized using regular expressions AS assign a schema to incoming data Assigns names and types to fields ( alias:type) (name:chararray, age:int, gpa:float) 8

9 records = LOAD '/input/excite-small.log USING PigStorage() AS (userid:chararray, timestamp:long, query:chararray); int Signed 32-bit integer 10 long Signed 64-bit integer 10L or 10l float 32-bit floating point 10.5F or 10.5f double 64-bit floating point 10.5 or 10.5e2 or 10.5E2 chararray Character array (string) in Unicode UTF-8 bytearray Byte array (blob) hello world tuple An ordered set of fields (T: tuple (f1:int, f2:int)) bag A collection of tuples (B: bag {T: tuple(t1:int, t2:int)}) Pig Latin Diagnostic Tools Display the structure of the Bag grunt> DESCRIBE <bag_name>; Display Execution Plan Produces Various reports, e.g. logical plan, MapReduce plan grunt> EXPLAIN <bag_name>; Illustrate how Pig engine transforms the data grunt> ILLUSTRATE <bag_name>; 9

10 FOREACH statement FOREACH <bag> GENERATE <data> Iterate over each element in the bag and produce a result grunt> records = LOAD /input/a.txt' AS (c:chararray, i:int); grunt> dump records; (a,1) (d,4) (c,9) (k,6) grunt> counts = foreach records generate i; grunt> dump counts; (1) (4) (9) (6) Joining Two Data Sets Join Steps Load records into a bag from input #1 Load records into a bag from input #2 Join the 2 data-sets (bags) by provided join key Default Join is Inner Join Rows are joined where the keys match Rows that do not have matches are not included in the result Inner join Set 1 Set 2 10

11 Simple join example 1. Load records into a bag from input #1 posts = load '/input/user-posts.txt' using PigStorage(',') as (user:chararray, post:chararray, date:long); 2. Load records into a bag from input #2 likes = load '/input/user-likes.txt' using PigStorage(',') as (user:chararray,likes:int,date:long); 3. Join the data sets when a key is equal in both data-sets then the rows are joined into a new single row; In this case when user name is equal userinfo = join posts by user, likes by user; dump userinfo; $ hdfs dfs -cat /input/user-posts.txt user1,funny Story, user2,cool Deal, user4,interesting Post, user5,yet Another Blog, $ hdfs dfs -cat /input/user-likes.txt user1,12, user2,7, user3,0, user4,50, $ pig /code/innerjoin.pig (user1,funny Story, ,user1,12, ) (user2,cool Deal, ,user2,7, ) (user4,interestingpost, ,user4,50, ) 11

12 Join re-uses the names of the input fields and prepends the name of the input bag <bag_name>::<field_name> grunt> describe posts; posts: {user: chararray,post: chararray,date: long} grunt> describe likes; likes: {user: chararray,likes: int,date: long} grunt> describe userinfo; UserInfo: { posts::user: chararray, posts::post: chararray, posts::date: long, likes::user: chararray, likes::likes: int, likes::date: long} Outer Join Records which will not join with the other record-set are still included in the result Left Outer Records from the first data-set are included whether they have a match or not. Fields from the unmatched (second) bag are set to null. Right Outer The opposite of Left Outer Join: Records from the second dataset are included no matter what. Fields from the unmatched (first) bag are set to null. Full Outer Records from both sides are included. For unmatched records the fields from the other bag are set to null. 12

13 Pig Use cases Loading large amounts of data Pig is built on top of Hadoop -> scales with the number of servers Alternative to manual bulkloading Using different data sources, e.g. collect web server logs, use external programs to fetch geo-location data for the users IP addresses, join the new set of geo-located web traffic to click maps stored Support for data sampling Hive Data Warehousing Solution built on top of Hadoop Provides SQL-like query language named HiveQL Minimal learning curve for people with SQL expertise Data analysts are target audience Early Hive development work started at Facebook in 2007 Translates HiveQL statements into a set of MapReduce Jobs which are then executed on a Hadoop Cluster 13

14 Hive Ability to bring structure to various data formats Simple interface for ad hoc querying, analyzing and summarizing large amounts of data Access to files on various data stores such as HDFS and HBase Hive does NOT provide low latency or realtime queries Even querying small amounts of data may take minutes Designed for scalability and ease-of-use rather than low latency responses Hive To support features like schema(s) and data partitioning Hive keeps its metadata in a Relational Database Packaged with Derby, a lightweight embedded SQL DB Default Derby based is good for evaluation an testing Schema is not shared between users as each user has their own instance of embedded Derby Stored in metastore_db directory which resides in the directory that hive was started from Can easily switch another SQL installation such as MySQL 14

15 Hive Architecture Hive Interface Options Command Line Interface (CLI) Hive Web Interface Java Database Connectivity (JDBC) Re-used from Relational Databases Database: Set of Tables, used for name conflict resolution Table: Set of Rows that have the same schema (same columns) Row: A single record; a set of columns Column: provides value and type for a single value 15

16 Hive creating a table hive> CREATE TABLE posts (user STRING, post STRING, time BIGINT) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY ',' > STORED AS TEXTFILE; OK Time taken: seconds creates a table with 3 columns How the underlying file should be parsed hive> show tables; OK posts Time taken: seconds hive> describe posts; OK user string post string time bigint Time taken: seconds Display schema for posts table Hive Query Data hive> select * from posts where user="user2"; OK user2 Cool Deal Time taken: seconds hive> select * from posts where time<= limit 2; OK user1 Funny Story user2 Cool Deal Time taken: seconds hive> 16

17 Partitions To increase performance Hive has the capability to partition data The values of partitioned column divide a table into segments Entire partitions can be ignored at query time Similar to relational databases indexes but not as granular Partitions have to be properly crated by users When inserting data must specify a partition At query time, whenever appropriate, Hive will automatically filter out partitions Bucketing Mechanism to query and examine random samples of data Break data into a set of buckets based on a hash function of a "bucket column" Capability to execute queries on a sub-set of random data Doesn t automatically enforce bucketing User is required to specify the number of buckets by setting # of reducer 17

18 Joins Hive support outer joins left, right and full joins Can join multiple tables Default Join is Inner Join Rows are joined where the keys match Rows that do not have matches are not included in the result Pig vs. Hive vs. Spark Hive Uses an SQL like query language called HQL Gives non-programmers the ability to query and analyze data in Hadoop. Pig Uses a workflow driven scripting language Don't need to be an expert Java programmer but need a few coding skills. Can be used to convert unstructured data into a meaningful form. Spark Successor to map-reduce in Hadoop, Emphasis on in-memory computing, lower level programming required than in Hive and Spark 18