Amazon EC2 KAIST Haejoon LEE

Transcription

1 Before VM Copy 계정 생성 및 sudo 권한, Java, Scala 설치 Hadoop Configuration 설정 Tar 압축 및 전송 >>tar cvfpz hadoop.tar.gz./hadoop >>scp -rp hadoop.tar.gz SecondaryNode:/usr/local >>scp 게./hadoop Tar 압축 해제 >>tar xvfzp hadoop.tar.gz Amazon EC2 KAIST Haejoon LEE Amazon First Setting on the website) Setting Generation / Security Group / Make one node as a MASTER / Choose Options of HDD and EBS 32GB / All ICMP, HTTP, SSH, All TCP as anywhere / Combine Available zones / PEM key / sudo apt-get install locale, sudo locale-gen en_us.utf-8, sudo locale-gen ko_kr.utf-8, locale (To use.pem file / chmod 400 kaist_jjoon.pem) First of all, you should install all program you need in the MASTER NODE: First account- ubuntu ->sudo adduser jjoon sudo, java, build-essential, scala, /etc/fstab/, ssh, ssh-keygen t rsa, opensshserver LABEL=cloudimg-rootfs / ext4 defaults,discard 0 0 /dev/xvdb /hdfs-1 auto defaults,nobootwait,comment=cloudconfig 0 2 /dev/xvdc /hdfs-2 auto defaults,nobootwait,comment=cloudconfig 0 2 /dev/xvdd /hdfs-3 auto defaults,nobootwait,comment=cloudconfig 0 2 /dev/xvde /hdfs-4 auto defaults,nobootwait,comment=cloudconfig 0 2

2 Secondly, make an image of the Master as AMIS ================================================ So far, One Node Job Thirdly, through the above image, launch 49 nodes by copying the settings we did at first. Set a view where you can only see Private IPs of NODES. Drag them and Paste them on excel to make csv file and /etc/hosts/ #!/usr/bin/python ID = 'jjoon' HOME = '/home/jjoon/' hosts_file = open (HOME + 'hosts') hosts = hosts_file.readlines() for host in hosts: comm = 'cat ' + HOME + '.ssh/id_rsa.pub ssh ' + ID + '@' + host.strip() + " 'cat >.ssh/authorized_keys'" hosts_file.close() Via this script, spread and collect keys from Nodes to Master After this ( type yes and password 49 times, you did collect public key and add known_hosts in.ssh folder. Then run this script #!/usr/bin/python HOME = '/home/jjoon/' FILE = '/home/jjoon/.ssh/known_hosts' KEY = '/home/ubuntu/jjoon.pem'

3 for host in hosts: comm = "scp "+ FILE + " " + host.strip() + ":.ssh/known_hosts" #!/usr/bin/python for /etc/hosts Spreading!!! HOME = '/home/jjoon/scripts/' FILE = '/etc/hosts' comm = 'cp /etc/hosts ~/' for host in hosts: comm = "scp ~/hosts " + host.strip() + ":~/hosts" comm = "ssh " + host.strip() + " 'sudo cp ~/hosts /etc/hosts'" Just for your information, do you know why we redirect /root file/ to the local disk because we cannot directly send it to all nodes. Thus, the script is like the above table. Due to some reasons, the mount files of hdfs-1, hdfs-2, hdfs-3, hdfs-4 could be root authority. And you should change that and send it to all nodes for saving partition file from Hadoop. #!/usr/bin/python BY USING UBUNTU ACCOUNT

4 HOME = '/home/jjoon/' FILE = '/etc/fstab' KEY = '/home/ubuntu/jjoon.pem' comm = 'cp /etc/fstab ~/' for host in hosts: comm = "scp -i " + KEY + " ~/fstab " + host.strip() + ":~/fstab" comm = "ssh -i " + KEY + " " + host.strip() + " 'sudo cp ~/fstab /etc/fstab'" #!/usr/bin/python BY USING UBUNTU ACCOUNT HOME = '/home/jjoon/' FILE = '/etc/hosts' KEY = '/home/ubuntu/jjoon.pem' comm = 'cp /etc/hosts ~/'

5 for host in hosts: comm = "ssh -i " + KEY + " " + host.strip() + " 'sudo chown jjoon:jjoon -R /hdfs-4'" Parser making csv file to tap file, actually you can use it on excel instead of this java file. import java.io.bufferedreader; import java.io.bufferedwriter; import java.io.filereader; import java.io.filewriter; import java.io.ioexception; import java.util.stringtokenizer; public class Tap { public static void main(string args[]) throws IOException { for(string arg: args) System.out.println(arg); String input = ""; String inputpath = args[0]; String outputpath = args[1]; int num = 0; int nodes = 0; String basicpath = Tap.class.getResource("").getPath(); System.out.println(basicPath); FileReader fr; FileWriter fw; BufferedReader br; BufferedWriter bw; fr = new FileReader(basicPath+inputPath); br = new BufferedReader(fr); fw = new FileWriter(basicPath+outputPath+".csv",true); bw = new BufferedWriter(fw); while((input = br.readline())!= null) { if(!(input.contains("#"))){ StringTokenizer token; int row; int i = 0; int j = 0; token = new StringTokenizer(input,"\t");

6 br.close(); bw.close(); fr.close(); fw.close(); while(token.hasmoretokens()) { String att = token.nexttoken(); bw.write(att); if(i == 0) bw.write(','); i++; bw.newline(); i = 0; Reference Video, actually if you use FTP, it must be easy!