Amazon EC2 Tutorial on Windows 7

Transcription

1 Amazon EC2 Tutorial on Windows 7 Created: Author: Hyun Kim Last Updated: Version Number: 0.1 Contact info: hyunk@loganbright.com Krish@loganbriht.com

2 In this tutorial, our ultimate goal is to use Amazon EC2 to install hadoop using 4 instances. We are going to give each instance different role, namely namenode, resoucemanager, slavenode(datanode and nodemanager), and secondarynode, and run current stable Hadoop as a ready production server. Let s get started. Let s start setting up EC2 go to Create an account Click on EC2 Create Instance (Your screen might be different since I ve already created instances. However, you should still be able to see the Create Instance button)

3 Select AMI I chose Ubuntu (HVM) Select instance that is freely usable

4 Configure Instance You can create one instance, install hadoop and then clone the instance. This is convenient since you don t have to set up hadoop on each instance. Add Storage Currently, 8gb is the lowest size I can go with. So I selected that one. Since we will be needing at least 4 nodes, it is ideal to keep the size 5-7 but since 8 is the minimum, we will create an instance with 8gb. Tag Instance Add key and value as you wish.

5 Add Rule ssh, all tcp, all icmp. Also set their sources to Anywhere for this tutorial. Click Review and Launch. After you launch your instance, you will see something similar to the picture below. (Oh, you can name your private key pair whatever you wish to. Just save it somewhere you know how to locate to because we will have to import it on puttygen very soon.) NOTE:DO NOT lose this file since you won t be able to access to your

6 instance without it. (Since you created only one node, you will see one node only.) Do copy and paste Public DNS and Private IPs on notepad and save it. You will be using these throughout the tutorial. Let s download some software to access to the instance. I m going to get PuTTY and PuTTYgen since they are quite easy to use. Download PuTTYgen and run it.

7 Click Import key I left passphrase blank to make it easier. You can set passphrase if you are worried about security. If you click Save private key

8 Click Yes and save the private key. The file will be in ppk format. Now let s access to the instance we created. In Host Name slot, enter your Public DNS there. You can find it here On putty go to SSH and Auth

9 Click Browse and select on the private key that we generated using puttygen. Save the session so that we don t have to do this everytime we try to access to instant and open it. Now we are ready to install hadoop. $sudo apt-get update

10 $sudo apt-get install openjdk-7-jdk Java installed. $nano ~/.bashrc Add the JAVA_HOME below at the top of ~/.bashrc # Set Hadoop-related environment variables export HADOOP_HOME=$HOME/hadoop export HADOOP_CONF_DIR=$HOME/hadoop/etc/hadoop export HADOOP_MAPRED_HOME=$HOME/hadoop export HADOOP_COMMON_HOME=$HOME/hadoop export HADOOP_HDFS_HOME=$HOME/hadoop export YARN_HOME=$HOME/hadoop #JAVA_HOME export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/ #PATH export PATH=$PATH:$HOME/hadoop/bin

11 To set the changes $source ~/.bashrc $cd $wget tar.gz $tar -xvf hadoop tar.gz For convenience, we will name the hadoop to hadoop $mv hadoop hadoop Configure SSH keypair

12 Do the following on master nodes only Open WinSCP Upload the private key file that was generated by puttygen I uploaded my private pair-key under /home/ubuntu $mkdir ~/.ssh/id_rsa Created the directory

13 $cp lbbdkeypair.ppk ~/.ssh/id_rsa Successfully added! Do this on your other master node. Passwordless SSH On master nodes, $ssh-agent bash $ssh-add However, these commands gave me this error

14 Let s reset permission back to default $sudo chmod 600 ~/.ssh/id_rsa $ssh-add Added successfully! do the above on each machine Let s edit slaves file on master nodes! $cd $nano hadoop/etc/hadoop/slaves Add hostname of your own slave node or nodes. Disable IPv6 $sudo nano /etc/sysctl.conf Add these at the bottom of the file # disable ipv6 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 Disable IPv6 on all the machines

15 edit /etc/hosts $sudo nano /etc/hosts Edit host name $sudo nano /etc/hostname

16 Copy and paste your namenode Public DNS. Edit hadoop-env.sh $nano hadoop/etc/hadoop/hadoop-env.sh export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

17 Edit core-site.xml $cd $nano hadoop/etc/hadoop/core-site.xml <property> <name>fs.defaultfs</name> <value>hdfs://localhost:9000</value> </property> Edit hdfs-site.xml <property> <name>dfs.replication</name> <value>1</value> </property>

18 <property> <name>dfs.namenode.name.dir</name> <value>file:/data/namenode</value> </property> <property> <name>dfs.datanode.name.dir</name> <value>file:/data/datanode</value> </property> Create storage for namenode and datanode $sudo mkdir /data/datanode

19 $sudo mkdir /data/namenode Change permission to these files $sudo chown -R ubuntu:ubuntu /data/datanode $sudo chown -R ubuntu:ubuntu /data/namenode Edit mapred-site.xml <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>

20 save it as mapred-site.xml from mapred-site.xml.template Edit yarn-site.xml <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.shufflehandler</value> </property>

21 Start DFS services $hadoop namenode -format Start all the daemons

22 $cd $cd hadoop/sbin $./hadoop-daemon.sh start namenode $./hadoop-daemon.sh start dataenode

23 $./yarn-daemon.sh start resourcemanager $./yarn-daemon.sh start nodemanager $./mr-jobhistory-daemon.sh start historyserver Open your favorite web browser and see if everything is running correctly

24 JobHistory

25 Installation of Single node Hadoop Completed Add Key Pair identity to SSH profile $cd ~/.ssh $chmod 644 authorized_keys $chmod 400 lbbdkeypair.pem $eval ssh-agent -s optionally in case it didn t work. try the command below after you try ssh-add first $exec ssh-agent bash $ssh-add lbbdkeypair.pem

26 $ssh Edit xml files

27 $cd $cd /hadoop/etc/hadoop core-site.xml edit hdfs-site.xml

28 yarn-site.xml clear previous files in /data/namenode $rm -rf /data/namenode/* $hadoop namenode -format

29 Run the command above until it asks you to press Y to format. $cd hadoop/sbin $./hadoop-daemon.sh start namenode namenode started let s configure resource manager node core-site.xml

30 hdfs-site.xml this file will be blank mapred-site.xml <property>

31 <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapred.job.trakcer</name> <value>ec us-west-2.compute.amazonaws.com:9001</value> </property> <property> <name>mapred.jobtrakcer.taskscheduler</name> <value>org.apache.hadoop.mapred.capacitytaskscheduer</value> </property> let s start resource manager $cd $cd hadoop/sbin $./yarn-daemon.sh start resourcemanager

32 Now let s run a slave node core-site.xml hdfs-site.xml

33 let s empty /data/datanode folder $rm -rf /data/datanode/* mapred-site.xml $sudo mkdir /data/datanode

34 $rm -rf /data/namenode $sudo chown -R ubuntu:ubuntu /data/datanode $cd hadoop/sbin $./hadoop-daemon.sh start datanode

35

36 It has a live node,which is shown in the picture below. It s node is ec us-west-2.compute.amazonaws.com, which is the slave node. on slave node, let s start node manager $cd

37 $cd hadoop/sbin $./yarn-daemon.sh start nodemanager All our daemons are running!