Hadoop MultiNode Cluster Setup

Transcription

1 Hadoop MultiNode Cluster Setup Sunil Raiyani Jayam Modi June 7, 2014 Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

2 Outline 4 Starting Daemons 1 Pre-Requisites 2 Network Settings 5 Map Reduce Task 6 References 3 Conguration Files Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

3 Pre-Requisites Pre-Requisites Setup a Hadoop single node cluster on the master and slaves as described in [2]. Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

4 Network Settings Network Settings To run a multinode cluster ensure that the master and all the slaves are on a single network. Identify the ip address of each system. Now make entries in the /etc/hosts le as follows: master slave01 These entries should also be in all the systems. Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

5 Network Settings SSH Login for Slaves SSH Login for Slaves Add the public key of master to all slaves using the command: ssh-copy-id -i $HOME/.ssh/id_dsa.pub Now ssh to the master and slaves for ensuring that passwordless ssh has been setup properly. ssh master ssh slave01 Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

6 Conguration Files Conguration Files Add the following lines bewteen <conguration> and < /conguration> tags to the les in $HADOOP_HOME/etc/hadoop folder for both master and slave [1]: core-site.xml <property> <name>fs.defaultfs</name> <value>hdfs://master:9000</value> </property> Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

7 Conguration Files Conguration Files Conguration Files yarn-site.xml <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.shufflehandler</value> </property> mapred-site.xml <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

8 Conguration Files Conguration Files Conguration Files hdfs-site.xml <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/home/hduser/mydata/hdfs/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/home/hduser/mydata/hdfs/namenode</value> </property> Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

9 Conguration Files Conguration Files Conguration Files Now add the following names of all slaves to the the $HADOOP_HOME/etc/slaves le. nano $HADOOP_HOME/etc/slaves slave01 Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

10 Starting Daemons Starting Daemons Format the namenode if you want to erase data on the Hadoop File System using the command hdfs namenode -format Run the following two scripts on the master node to start the hadoop and yarn daemons: start-dfs.sh start-yarn.sh Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

11 Starting Daemons Starting Daemons Starting Daemons To test of if the Daemons have started properly or not run the jps on the master and slave: Master: 9412 SecondaryNameNode 9784 NameNode Jps ResourceManager Slave: Datanode Nodemanager Jps jps jps Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

12 Map Reduce Task Map Reduce task Run the following commands on the master system to run a sample wordcount program on the cluster: sudo mkdir /in sudo nano /in/file Type in some text and save the le. hdfs dfs -copyfromlocal /in/file /file yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples jar wordcount /file /out Note : The /out directory must not already exist on the HDFS system else it will give an error. Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

13 Map Reduce Task Map Reduce Task Map Reduce Task The output of the mapreduce task will be saved in the /out directory on the distributed system. Use the follwing command to view the result : hdfs dfs -text /out/part-r Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14

14 References References I [1] accessed on June 7, 2014 [2] Hadoop Installation Manual : /les/ P1_saatvik14_Week_2_HadoopHiveInstallation_1_ 2014_05_17.pdf Sunil Raiyani Jayam Modi Hadoop MultiNode Cluster Setup June 7, / 14