Pig vs Hive. Big Data 2014

Transcription

1 Pig vs Hive Big Data 2014

2 Pig Configuration In the bash_profile export all needed environment variables

3 Pig Configuration Download a release of apache pig: pig tar.gz

4 Pig Configuration Go to the conf directory in the pig-home directory rename the file pig.properties.template in pig.properties

5 Pig Running Running Pig: $:~pig-*/bin/pig <parameters> Try the following command, to get a list of Pig commands: $:~pig-*/bin/pig -help Run modes: local $:~pig-*/bin/pig -x local! mapreduce $:~pig-*/bin/pig or $:~pig-*/bin/pig -x mapreduce

6 Pig in Local Running Pig in Local: $:~pig-*/bin/pig -x local Grunt Shell: grunt> A = LOAD 'passwd' using PigStorage(':'); grunt> B = FOREACH A GENERATE $0 as id; grunt> dump B; grunt> store B; Script file: $:~pig-*/bin/pig -x local myscript.pig

7 Pig in Local: Examples Word Count using Pig Count words in a text file separated by lines and spaces Basic idea: Load this file using a loader Foreach record generate word token Group by each word Count number of words in a group Store to file words.txt program program pig pig program pig hadoop pig latin latin

8 Pig in Local: Examples Word Count using Pig $:~pig-*/bin/pig -x local! grunt> myinput = LOAD <myhome>/words.txt USING TextLoader() as (myword:chararray); grunt> words = FOREACH myinput GENERATE FLATTEN(TOKENIZE(*)); grunt> grouped = GROUP words BY $0; grunt> counts = FOREACH grouped GENERATE group, COUNT(words); grunt> store counts into '<myhome>/pigoutput' using PigStorage();

9 Pig in Local: Examples Word Count using Pig $:~pig-*/bin/pig -x local wordcount.pig wordcount.pig myinput = LOAD <myhome>/words.txt USING TextLoader() as (myword:chararray); words = FOREACH myinput GENERATE FLATTEN(TOKENIZE(*)); grouped = GROUP words BY $0; counts = FOREACH grouped GENERATE group, COUNT(words); store counts into '<myhome>/pigoutput' using PigStorage();

10 Pig in Local: Examples Word Count using Pig myinput = LOAD <myhome>/words.txt USING TextLoader() as (myword:chararray); words = FOREACH myinput GENERATE FLATTEN(TOKENIZE(*)); grouped = GROUP words BY $0; counts = FOREACH grouped GENERATE group, COUNT(words); store counts into '<myhome>/pigoutput' using PigStorage();

11 Pig in Local: Examples Word Count using Pig myinput = LOAD <myhome>/words.txt USING TextLoader() as (myword:chararray); words = FOREACH myinput GENERATE FLATTEN(TOKENIZE(*)); grouped = GROUP words BY $0; counts = FOREACH grouped GENERATE group, COUNT(words); store counts into '<myhome>/pigoutput' using PigStorage();

12 Pig in Local: Examples Word Count using Pig myinput = LOAD <myhome>/words.txt USING TextLoader() as (myword:chararray); words = FOREACH myinput GENERATE FLATTEN(TOKENIZE(*)); grouped = GROUP words BY $0; counts = FOREACH grouped GENERATE group, COUNT(words); store counts into '<myhome>/pigoutput' using PigStorage();

13 Pig in Local: Examples Word Count using Pig myinput = LOAD <myhome>/words.txt USING TextLoader() as (myword:chararray); words = FOREACH myinput GENERATE FLATTEN(TOKENIZE(*)); grouped = GROUP words BY $0; counts = FOREACH grouped GENERATE group, COUNT(words); store counts into '<myhome>/pigoutput' using PigStorage();

14 Pig in Local: Examples Word Count using Pig myinput = LOAD <myhome>/words.txt USING TextLoader() as (myword:chararray); words = FOREACH myinput GENERATE FLATTEN(TOKENIZE(*)); grouped = GROUP words BY $0; counts = FOREACH grouped GENERATE group, COUNT(words); store counts into '<myhome>/pigoutput' using PigStorage();

15 Pig in Local: Examples Word Count using Pig the directory '<myhome>/pigoutput' has not to exist before the execution of the script myinput = LOAD <myhome>/words.txt USING TextLoader() as (myword:chararray); words = FOREACH myinput GENERATE FLATTEN(TOKENIZE(*)); grouped = GROUP words BY $0; counts = FOREACH grouped GENERATE group, COUNT(words); store counts into '<myhome>/pigoutput' using PigStorage();

16 Pig in MapReduce: Examples Word Count using Pig $:~hadoop-*/bin/hadoop dfs -mkdir input $:~hadoop-*/bin/hadoop dfs -copyfromlocal /tmp/words.txt input $:~pig-*/bin/pig -x mapreduce wordcountmr.pig wordcountmr.pig myinput = LOAD input/words.txt USING TextLoader() as (myword:chararray); words = FOREACH myinput GENERATE FLATTEN(TOKENIZE(*)); grouped = GROUP words BY $0; counts = FOREACH grouped GENERATE group, COUNT(words); store counts into 'pigoutput' using PigStorage();

17 Pig in MapReduce: Examples Word Count using Pig hdfs:input $:~pig-*/bin/pig -x mapreduce wordcountmr.pig

18 Pig in MapReduce: Examples Word Count using Pig hdfs:pigoutput part-r-00000

19 Pig in Local: Examples Computing average number of page visits by user Logs of user visiting a webpage consists of (user,url,time) Fields of the log are tab separated and in text format Basic idea: Load the log file Group based on the user field Count the group Calculate average for all users Visualize result visits.log user url time Amy 8:00 Amy 8:05 Amy 10:00 Amy 10:05 Fred cnn.com/index.htm 12:00 Fred cnn.com/index.htm 13:00

20 Pig in Local: Examples Computing average number of page visits by user $:~pig-*/bin/pig -x local average_visits_log.pig average_visits_log.pig visits = LOAD <myhome>/visits.log as (user,url,time); user_visits = GROUP visits BY user; user_cnts = FOREACH user_visits GENERATE group as user, COUNT(visits) as numvisits; all_cnts = GROUP user_cnts all; avg_cnt = FOREACH all_cnts GENERATE AVG(user_cnts.numvisits); dump avg_cnt;

21 Pig in Local: Examples Computing average number of page visits by user visits = LOAD <myhome>/visits.log as (user,url,time); user_visits = GROUP visits BY user; user_cnts = FOREACH user_visits GENERATE group as user, COUNT(visits) as numvisits; all_cnts = GROUP user_cnts all; avg_cnt = FOREACH all_cnts GENERATE AVG(user_cnts.numvisits); dump avg_cnt;

22 Pig in Local: Examples Computing average number of page visits by user visits = LOAD <myhome>/visits.log as (user,url,time); user_visits = GROUP visits BY user; user_cnts = FOREACH user_visits GENERATE group as user, COUNT(visits) as numvisits; all_cnts = GROUP user_cnts all; avg_cnt = FOREACH all_cnts GENERATE AVG(user_cnts.numvisits); dump avg_cnt;

23 Pig in Local: Examples Computing average number of page visits by user visits = LOAD <myhome>/visits.log as (user,url,time); user_visits = GROUP visits BY user; user_cnts = FOREACH user_visits GENERATE group as user, COUNT(visits) as numvisits; all_cnts = GROUP user_cnts all; avg_cnt = FOREACH all_cnts GENERATE AVG(user_cnts.numvisits); dump avg_cnt;

24 Pig in Local: Examples Computing average number of page visits by user visits = LOAD <myhome>/visits.log as (user,url,time); user_visits = GROUP visits BY user; user_cnts = FOREACH user_visits GENERATE group as user, COUNT(visits) as numvisits; all_cnts = GROUP user_cnts all; avg_cnt = FOREACH all_cnts GENERATE AVG(user_cnts.numvisits); dump avg_cnt;

25 Pig in Local: Examples Computing average number of page visits by user visits = LOAD <myhome>/visits.log as (user,url,time); user_visits = GROUP visits BY user; user_cnts = FOREACH user_visits GENERATE group as user, COUNT(visits) as numvisits; all_cnts = GROUP user_cnts all; avg_cnt = FOREACH all_cnts GENERATE AVG(user_cnts.numvisits); dump avg_cnt;

26 Pig in Local: Examples Computing average number of page visits by user visits = LOAD <myhome>/visits.log as (user,url,time); user_visits = GROUP visits BY user; user_cnts = FOREACH user_visits GENERATE group as user, COUNT(visits) as numvisits; all_cnts = GROUP user_cnts all; avg_cnt = FOREACH all_cnts GENERATE AVG(user_cnts.numvisits); dump avg_cnt;

27 Pig in Local: Examples Identify users who visit Good Pages Good pages are those pages visited by users whose page rank is greater than 0.5 Basic idea: Join table based on url Group based on user Calculate average page rank of user visited pages Filter user who has average page rank greater than 0.5 Store the result visits.log user url time Amy 8:00 Amy 8:05 Amy 10:00 Amy 10:05 Fred cnn.com/index.htm 12:00 Fred cnn.com/index.htm 13:00 url pages.log pagerank

28 Pig in Local: Examples Identify users who visit Good Pages $:~pig-*/bin/pig -x local good_users.pig good_users.pig visits = LOAD <myhome>/visits.log as (user:chararray,url:chararray,time:chararray); pages = LOAD <myhome>/pages.log as (url:chararray,pagerank:float); visits_pages = JOIN visits BY url, pages BY url; user_visits = GROUP visits_pages BY user; user_avgpr = FOREACH user_visits GENERATE group, AVG(visits_pages.pagerank) as avgpr; good_users = FILTER user_avgpr BY avgpr > 0.5f; store good_users into <myhome>/pigoutput ;

29 Pig in Local: Examples Identify users who visit Good Pages Load files for processing with appropriate types visits = LOAD <myhome>/visits.log as (user:chararray,url:chararray,time:chararray); pages = LOAD <myhome>/pages.log as (url:chararray,pagerank:float); visits_pages = JOIN visits BY url, pages BY url; user_visits = GROUP visits_pages BY user; user_avgpr = FOREACH user_visits GENERATE group, AVG(visits_pages.pagerank) as avgpr; good_users = FILTER user_avgpr BY avgpr > 0.5f; store good_users into <myhome>/pigoutput ;

30 Pig in Local: Examples Identify users who visit Good Pages visits = LOAD <myhome>/visits.log as (user:chararray,url:chararray,time:chararray); pages = LOAD <myhome>/pages.log as (url:chararray,pagerank:float); visits_pages = JOIN visits BY url, pages BY url; user_visits = GROUP visits_pages BY user; user_avgpr = FOREACH user_visits GENERATE group, AVG(visits_pages.pagerank) as avgpr; good_users = FILTER user_avgpr BY avgpr > 0.5f; store good_users into <myhome>/pigoutput ;

31 Pig in Local: Examples Identify users who visit Good Pages visits = LOAD <myhome>/visits.log as (user:chararray,url:chararray,time:chararray); pages = LOAD <myhome>/pages.log as (url:chararray,pagerank:float); visits_pages = JOIN visits BY url, pages BY url; user_visits = GROUP visits_pages BY user; user_avgpr = FOREACH user_visits GENERATE group, AVG(visits_pages.pagerank) as avgpr; good_users = FILTER user_avgpr BY avgpr > 0.5f; store good_users into <myhome>/pigoutput ;

32 Pig in Local: Examples Identify users who visit Good Pages visits = LOAD <myhome>/visits.log as (user:chararray,url:chararray,time:chararray); pages = LOAD <myhome>/pages.log as (url:chararray,pagerank:float); visits_pages = JOIN visits BY url, pages BY url; user_visits = GROUP visits_pages BY user; user_avgpr = FOREACH user_visits GENERATE group, AVG(visits_pages.pagerank) as avgpr; good_users = FILTER user_avgpr BY avgpr > 0.5f; store good_users into <myhome>/pigoutput ;

33 Pig in Local: Examples Identify users who visit Good Pages visits = LOAD <myhome>/visits.log as (user:chararray,url:chararray,time:chararray); pages = LOAD <myhome>/pages.log as (url:chararray,pagerank:float); visits_pages = JOIN visits BY url, pages BY url; user_visits = GROUP visits_pages BY user; user_avgpr = FOREACH user_visits GENERATE group, AVG(visits_pages.pagerank) as avgpr; good_users = FILTER user_avgpr BY avgpr > 0.5f; store good_users into <myhome>/pigoutput ;

34 Pig in Local: Examples Identify users who visit Good Pages visits = LOAD <myhome>/visits.log as (user:chararray,url:chararray,time:chararray); pages = LOAD <myhome>/pages.log as (url:chararray,pagerank:float); visits_pages = JOIN visits BY url, pages BY url; user_visits = GROUP visits_pages BY user; user_avgpr = FOREACH user_visits GENERATE group, AVG(visits_pages.pagerank) as avgpr; good_users = FILTER user_avgpr BY avgpr > 0.5f; store good_users into <myhome>/pigoutput ;

35 Pig in Local with User Defined Functions: Examples Find all planets similar and closed to Earth Planets similar and closed to Earth are that with oxygen and whose distance from Earth is less than 5 Basic idea: Define a User Defined Function (UDF) Filter planets using UDF planets.txt planet, color, atmosphere, distanceformearth gallifrey, blue, oxygen, skaro, blue, phosphorus, 10.5 krypton, red, oxygen, 2.5 apokolips, white, unknown, 0 klendathu, orange, oxygen, 0.89 asgard, unknown, unknown, 0 mars, yellow, carbon dioxide, thanagar, yellow, oxygen, 3.29 planet x, yellow, unknown, 0.78 warworld, red, phosphorus, 10.1 daxam, red, oxygen, 7.2 oa, blue white, nitrogen, 2.4 Gliese 667Cc, red dwarf, unknown, 22

36 Pig in Local with User Defined Functions: Examples Find all planets similar and closed to Earth DistanceFromEarth.java package myudfs; import java.io.ioexception; import org.apache.pig.filterfunc; import org.apache.pig.data.tuple;! public class DistanceFromEarth extends FilterFunc { public Boolean exec(tuple input) throws IOException { if (input == null input.size() == 0) return null; try { Object value = input.get(0); if (value instanceof Double) return ((Double)value) <5; } catch(exception ee) { throw new IOException("Caught exception processing input row ", ee); } return null; } }

37 Pig in Local with User Defined Functions: Examples Find all planets similar and closed to Earth PlanetWithOxygen.java package myudfs; import java.io.ioexception; import org.apache.pig.filterfunc; import org.apache.pig.data.tuple;! public class PlanetWithOxygen extends FilterFunc { public Boolean exec(tuple input) throws IOException { if (input == null input.size() == 0) return null; try { String value = (String)input.get(0); return (value.indexof("oxygen") >=0); } catch(exception ee) { throw new IOException("Caught exception processing input row ", ee); } } }

38 Pig in Local with User Defined Functions: Examples Find all planets similar and closed to Earth myudfs.jar

39 Pig in Local with User Defined Functions: Examples Find all planets similar and closed to Earth $:~pig-*/bin/pig -x local planets.pig planets.pig REGISTER <myhome>/myudfs.jar ; planets = LOAD <myhome>/planets.txt USING PigStorage(,') as (planet:chararray,color:chararray,atmosphere:chararray,distance:double); result = FILTER planets BY myudfs.planetwithoxygen(atmosphere) AND myudfs.distancefromearth(distance); store result into <myhome>/pigoutput ;

40 Pig in Local with User Defined Functions: Examples Find all planets similar and closed to Earth REGISTER <myhome>/myudfs.jar ; planets = LOAD <myhome>/planets.txt USING PigStorage(,') as (planet:chararray,color:chararray,atmosphere:chararray,distance:double); result = FILTER planets BY myudfs.planetwithoxygen(atmosphere) AND myudfs.distancefromearth(distance); store result into <myhome>/pigoutput ;

41 Pig in Local with User Defined Functions: Examples Sort employees by department and by stack ranking. Basic idea: name, stackrank, department Define a User Defined Function (UDF) order employees using UDF employees.txt JohnS, 9.5, Accounting Bill, 6, Marketing Franklin, 7, Engineering Marci, 8, Exec Joe DeAngel, 4.5, Finance Steve Francis, 9, Accounting Sam Shade, 6.5, Engineering Sandi, 9, Exec Roderick Trevers, 7, Accounting Terri DeHaviland, 8.5, Exec Colin McCullers, 8, Marketing Fay LaMore, 9, Marketing

42 Pig in Local with User Defined Functions: Examples Sort employees by department and by stack ranking. {(rank:int, name:chararray, stackrank:double, department:chararray)}") def enumerate_bag(input): output = [] for rank, item in enumerate(input): output.append(tuple([rank] + list(item))) return output rankudf.py

43 Pig in Local with User Defined Functions: Examples Sort employees by department and by stack ranking. $:~pig-*/bin/pig -x local employee.pig employee.pig REGISTER <myhome>/rankudf.py ; employees = LOAD <myhome>/employees.txt USING PigStorage(,') as (name:chararray, stackrank:double, department:chararray); employees_by_department = GROUP employees BY department; result = FOREACH employees_by_department{ sorted = ORDER employees BY stackrank desc; ranked = myudf.enumerate_bag(sorted); generate flatten(ranked); }; store result into <myhome>/pigoutput ;

44 Pig in Local with User Defined Functions: Examples Sort employees by department and by stack ranking. REGISTER <myhome>/rankudf.py ; employees = LOAD <myhome>/employees.txt USING PigStorage(,') as (name:chararray, stackrank:double, department:chararray); employees_by_department = GROUP employees BY department; result = FOREACH employees_by_department{ sorted = ORDER employees BY stackrank desc; ranked = myudf.enumerate_bag(sorted); generate flatten(ranked); }; store result into <myhome>/pigoutput ;

45 Hive Configuration In the bash_profile export all needed environment variables

46 Hive Configuration Translates HiveQL statements into a set of MapReduce jobs which are then executed on a Hadoop Cluster Execute on Hadoop Cluster HiveQL Hive Monitor/Report Client Machine Hadoop Cluster

47 Hive Configuration Download a binary release of apache Hive: hive bin.tar.gz

48 Hive Configuration In the conf directory of hive-home directory set hive-env.sh file set the HADOOP_HOME # Set HADOOP_HOME to point to a specific hadoop install directory HADOOP_HOME=/Users/mac/Documents/hadoop-1.2.1

49 Hive Configuration Hive uses Hadoop In addition, you must create /tmp and /user/hive/warehouse and set them chmod g+w in HDFS before you can create a table in Hive. Commands to perform this setup: $:~$HADOOP_HOME/bin/hadoop dfs -mkdir $:~$HADOOP_HOME/bin/hadoop dfs -mkdir $:~$HADOOP_HOME/bin/hadoop dfs -chmod g+w $:~$HADOOP_HOME/bin/hadoop dfs -chmod g+w /tmp /user/hive/warehouse /tmp /user/hive/warehouse

50 Hive Running Running Hive: $:~hive-*/bin/hive <parameters> Try the following command to acces to Hive shell: $:~hive-*/bin/hive Hive Shell Logging initialized using configuration in jar:file:/users/ mac/documents/hive bin/lib/hive-common jar!/ hive-log4j.properties Hive history Air-di-mac.local_ _ txt hive>

51 Hive Running In the Hive Shell you can call any HiveQL statement: create a table hive> CREATE TABLE pokes (foo INT, bar STRING); OK Time taken: seconds hive> CREATE TABLE invites (foo INT, bar STRING) PARTITIONED BY (ds STRING); OK Time taken: seconds browsing through Tables: lists all the tables hive> SHOW TABLES; OK invites pokes Time taken: seconds, Fetched: 2 row(s)

52 Hive Running browsing through Tables: lists all the tables that end with 's'. hive> SHOW TABLES.*s ; OK invites pokes Time taken: seconds, Fetched: 2 row(s) browsing through Tables: shows the list of columns of a table. hive> DESCRIBE invites; OK foo int None bar string None ds string None # Partition Information # col_name data_type comment ds string None Time taken: seconds, Fetched: 8 row(s)

53 Hive Running altering tables hive> ALTER TABLE events RENAME TO 3koobecaf; hive> ALTER TABLE pokes ADD COLUMNS (new_col INT); hive> ALTER TABLE invites ADD COLUMNS (new_col2 INT COMMENT 'a comment'); hive> ALTER TABLE invites REPLACE COLUMNS (foo INT, bar STRING, baz INT COMMENT 'baz replaces new_col2'); dropping Tables hive> DROP TABLE pokes;

54 Hive Running DML operations takes file from local file system hive> LOAD DATA LOCAL INPATH './examples/files/kv1.txt' OVERWRITE INTO TABLE pokes; takes file from HDFS file system hive> LOAD DATA INPATH /user/hive/files/kv1.txt OVERWRITE INTO TABLE pokes; SQL query hive> SELECT * FROM pokes;

55 Hive Configuration on the Job Tracker of Hadoop By default, Hive utilizes LocalJobRunner; Hive can use the JobTracker of Hadoop In the conf directory of hive-home directory you have to add and edit the hive-site.xml file

56 Hive Configuration on the Job Tracker of Hadoop In the conf directory of hive-home directory you have to add and edit the hive-site.xml file <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hive.exec.scratchdir</name> <value>/users/mac/documents/hive bin/scracth</value> <description>scratch space for Hive jobs</description> </property> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> <description>location of resourcemanager so Hive knows where to execute mapreduce Jobs</description> </property> </configuration>

57 Hive Running Running Hive One Shot command: $:~hive-*/bin/hive -e <command> For instance: $:~hive-*/bin/hive -e SELECT * FROM mytable LIMIT 3 Result OK name1 10 name2 20 name3 30

58 Hive Running Executing Hive queries from file: $:~hive-*/bin/hive -f <file> For instance: $:~hive-*/bin/hive -f query.hql query.hql SELECT * FROM mytable LIMIT 3

59 Hive Running Executing Hive queries from file inside the Hive Shell $:~ cat /path/to/file/query.hql SELECT * FROM mytable LIMIT 3 $:~hive-*/bin/hive hive> SOURCE /path/to/file/query.hql;

60 Hive in Local: Examples Word Count using Hive wordcounts.hql CREATE TABLE docs (line STRING); LOAD DATA LOCAL INPATH./exercise/data/words.txt' OVERWRITE INTO TABLE docs; CREATE TABLE word_counts AS SELECT word, count(1) AS count FROM (SELECT explode(split(line, '\s')) AS word FROM docs) w words.txt program program pig pig program pig hadoop pig latin latin GROUP BY word ORDER BY word;

61 Hive in Local: Examples Word Count using Hive words.txt $:~hive-*/bin/hive -f wordcounts.hql program program pig pig program pig hadoop pig latin latin

62 Hive in Local with User Defined Functions: Examples Convert unixtime to a regular time date format subscribers.txt name, department, , time Frank Black, 1001, frankdaman@eng.example.com, Jolie Guerms, 1006, jguerms@ga.example.com, Mossad Ali, 1001, mali@eng.example.com, Chaka Kaan, 1006, ckhan@ga.example.com, Verner von Kraus, 1007, verner@example.com, Lester Dooley, 1001, ldooley@eng.example.com, Basic idea: Define a User Defined Function (UDF) Convert time field using UDF

63 Hive in Local with User Defined Functions: Examples Convert unixtime to a regular time date format package com.example.hive.udf; Unix2Date.java! import java.util.date; import java.util.timezone; import java.text.simpledateformat; import org.apache.hadoop.hive.ql.exec.udf; import org.apache.hadoop.io.text;! public class Unix2Date extends UDF{ public Text evaluate(text text) { if(text == null) return null; long timestamp = Long.parseLong(text.toString()); // timestamp*1000 is to convert seconds to milliseconds Date date = new Date(timestamp*1000L); // the format of your date SimpleDateFormat sdf = new SimpleDateFormat("dd-MM-yyyy HH:mm:ss z"); sdf.settimezone(timezone.gettimezone("gmt+2")); String formatteddate = sdf.format(date); return new Text(formattedDate); } }

64 Hive in Local with User Defined Functions: Examples Convert unixtime to a regular time date format unix_date.jar

65 Hive in Local with User Defined Functions: Examples Convert unixtime to a regular time date format $:~hive-*/bin/hive -f time_conversion.hql time_conversion.hql CREATE TABLE IF NOT EXISTS subscriber ( username STRING, dept STRING, STRING, provisioned STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','; LOAD DATA LOCAL INPATH./exercise/data/subscribers.txt' INTO TABLE subscriber; add jar./exercise/jar_files/unix_date.jar; CREATE TEMPORARY FUNCTION unix_date AS 'com.example.hive.udf.unix2date'; SELECT username, unix_date(provisioned) FROM subscriber;

66 Hive in Local with User Defined Functions: Examples Convert unixtime to a regular time date format $:~hive-*/bin/hive -f time_conversion.hql Frank Black :36:00 GMT+02:00 Jolie Guerms :00:00 GMT+02:00 Mossad Ali Chaka Kaan :00:32 GMT+02: :32:02 GMT+02:00 Verner von Kraus :36:25 GMT+02:00 Lester Dooley :34:10 GMT+02:00 Time taken: 9.12 seconds, Fetched: 6 row(s)

67 Quickly Wrap Up! Two ways of doing one thing OR! One way of doing two things

68 Two ways of doing same thing Both generate map-reduce jobs from a query written in higher level language. Both frees users from knowing all the little secrets of Map- Reduce & HDFS.

69 Language PigLatin: Procedural data-flow language A = LOAD mydata ; dump A; HiveQL: Declarative SQLish language SELECT * FROM mytable ;

70 Different languages = Different users Pig: More popular among programmers researchers Hive: More popular among analysts

71 Different users = Different usage pattern Pig: programmers: Writing complex data pipelines researchers: Doing ad-hoc analysis typically employing Machine Learning Hive: analysts: Generating daily reports

72 Different usage pattern Different Usage Pattern Data Collection Data Factory Data Warehouse Data Data Collection Data Factory Data Data Warehouse Pig Pig Hive Hive -Pipeline Pipelines Iterative Iterative Processing Processing Research -Iterative Processing Research -Research BI BI Tools -BI Tools tools Analysis Analysis -Analysis 7 7 7

73 Different usage pattern = Different future directions Pig is evolving towards a language of its own Users are asking for better dev environment: debugger, linker, editor etc. Hive is evolving towards Data-warehousing solution Users are asking for better integration with other systems (O/JDBC)

74 Resources

75 Pig vs Hive Big Data 2014