project collects data from national events, both natural and manmade, to be stored and evaluated by

Transcription

1 Joseph Sebastian CS 2994 Spring 2014 Undergraduate Research Final Paper GOALS The goal of my research was to assist the Integrated Digital Event Archive (IDEAL) team in transferring their Twitter data on world events from a MySQL server into a Hadoop cluster. The IDEAL project collects data from national events, both natural and manmade, to be stored and evaluated by varying metrics. I shadowed a Ph. D Student, Sunshin Lee, in setting up the cluster and learning about open source tools to help me with my task. I learned about an Apache tool sqoop that uses Java Database Connectivity (JDBC) to transfer data between a relational database and several destinations within Hadoop (e.q. Hadoop Distributed File System and Hive). I spent my semester of research developing commands and methods for using this tool to accomplish my goal. PROCEDURE My first objective was to learn more about Hadoop and practice the sqoop commands off of the cluster. I did this with the help of Cloudera s free Virtual Machine which came pre- loaded with most of the tools that I needed. I had to install MySQL within the machine and import an IDEAL table to work with which contained data about sink holes. I then practiced inserting and deleting this dataset within Hadoop Distributed File System (HDFS) and Hive. With sufficient practice I then moved on to the 162 table IDEAL database that contained roughly 206 million tweets and began importing it into our cluster. The process was difficult since some of the larger datasets (containing almost 20 million tweets) could take up to an hour to import, and each job had to run one at a time. Due to this restriction I wrote a bash script that would import multiple tables sequentially in a single run, which proved to be very efficient.

2 I faced several problems while working on this project, first of which dealt with setting up the cluster to work with the sqoop tool. The node within the cluster which housed our SQL Server did not originally contain JDBC which was required to work with the tool and took some time debugging to detect. After installing the necessary packages the tool worked as expected and I was able to continue. I also encountered problem when trying to perform an incremental import on our data since I had troubles finding an adequate variable to merge multiple imports together with. This problem has yet to be resolved however a few solutions have been found. Conclusion Through my research this semester I gained experience with open source tools such as Hadoop and sqoop while working with a large team on a project. This was my first experience with large amounts of data and working on a cluster and I found it very interesting. At the end of the semester I had accomplished my goal of imported all 206 million tweets into the Distributed File System. Thanks

3 Undergraduate Research Final Presentation Braeden Sebastian

4 Goals Get all archived Tweets into Hadoop Distributed File System o Tweets are currently stored in a MySQL Server o Use Apache s sqoop tool to help transfer and partition the data into HDFS and Hive Work on developing appropriate sqoop commands to take data from the relational world and prepare it for Hadoop

5 What is Hadoop? Enables processing on large amounts of data Main components o Map Reduce o HDFS - Hadoop Distributed File System o Hive Relational Database Notably scalable and cost effective Used by major companies (facebook, twitter, yahoo)

6 Cloudera Virtual Machine Provides Hadoop functionality for use with sqoop

7 MySQL Installed MySQL on Cloudera virtual machine and imported an IDEAL dataset

8 What is sqoop?

9 What is HDFS? Hadoop Distributed File System Imported data is partitioned into small blocks and spread out amongst data nodes in a cluster These blocks can then be accessed used with Map Reduce jobs o Use of blocks provides great scalability for large amounts of data

10 Transferring Data into HDFS The below sqoop command will take a dataset from MySQL and put it into HDFS

11 Checking HDFS for Dataset This command will list the contents of HDFS along with all subdirectories

12 What is Hive? Hadoop s built in data warehouse Provides reduced SQL functionality through the Hive Query Language (HQL) o Commands are executed as Map Reduce jobs and sent across the cluster Hive queries are reliant Hadoop and Map Reduce which causing them significant latency

13 Transferring Data into Hive

14 Querying Hive This command prints the location of the Hive data within HDFS

15 Execution of Project Approximately 162 datasets in a YourTwapperKeeper collection Set up cluster to support sqoop Unrealistic to import sequentially by hand Accessing remotely does not work for large datasets o Large datasets took about an hour to import Wrote bash script to perform batch imports to be run directly on cluster

16 Problems Encountered Cluster did not originally work with sqoop Incremental imports cannot work cleanly with given data o Requires a merge key to combine imports o Problems automating incremental imports

17 Results Learned about open source tools like Hadoop and sqoop Working on a large scale, collaborative project Introduced into big data and high performance computing Successfully imported 206 million tweets into Hadoop Distributed File System