用企鵝龍打造多人雲端實驗叢集. Jazz Wang Yao-Tsung Wang. Building Multiuser Hadoop Testbed with DRBL. 1/44

Transcription

1 用企鵝龍打造多人雲端實驗叢集 Building Multiuser Hadoop Testbed with DRBL Jazz Wang Yao-Tsung Wang 1/44 1

2 Programmer v.s. System Admin. Source: 2/44 Source:

3 Agenda PART 1 : What is Cluster Computing? How to deploy PC cluster? PART 2 : What is DRBL and Clonezilla? Can DRBL help to deploy Hadoop? PART 3 : 3/44 Live Demo of DRBL Live and Clonezilla Live

4 PART 1 : PC Cluster 101 Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw 4/44

5 At First, We have PC Cluster It'd better be 2 n 5/44 Manage Scheduler

6 Then, We connect 5 PCs with Gigabit Ethernet Switch GiE Switch WAN 6/44 10/100/1000 MBps Add 1 NIC for WAN

7 Compute Nodes 4 Compute Nodes will communicate via LAN Switch. Only Manage Node LAN Switch have Internet Access for Security! WAN 7/44 Manage Node

8 Compute Nodes Basic System Setup for Cluster 8/44 Messaging MPICH GCC Account Mgnt. SSHD NIS GNU Libc Bash Perl Kernel Module Linux Kernel Boot Loader YP

9 On Manage Node, We need to install Scheduler and Network File System for sharing Files with Compute Node Job Mgnt. Messaging OpenPBS MPICH File Sharing GCC NFS Bash Perl Extra 9/44 Account Mgnt. SSHD NIS GNU Libc Kernel Module Linux Kernel Boot Loader YP

10 Research topics about PC Cluster System Architecture Cluster Computing Parallel Computing Parallel Algorithms And Applications 10/44 Process Architecture Storage Architecture Network Architecture System-level Middleware Share Memory Programming Distributed Memory Programming Application-level Middleware Programming Ref: Cluster Computing in the Classroom: Topics, Guidelines, and Experiences

11 Challenges of Cluster Computing 11/44 Hardware Ethernet Speed / PC Density Power / Cooling / Heat Network and Storage Architecture Software Job Scheduler ( Cluster level ) Account Management File Sharing / Package Management Limitation Shared Memory Global Memory Management

12 Common Method to deploy Cluster 3. Configure 2. Cloning to multiple machine Settings 4. Install Job Scheduler 1. Setup one Template machine 12/44 5. Running Benchmark

13 Challenges of Common Method? t n u o c c A r e s U w e N d d A Upgrade Softw are? How to share u ser dat a? n o i t a z i n o r c n y S n o i t a r u g i f n o C 13/44

14 How to deploy Nodes???? 資料標題 Scaling Hadoop to 4000 nodes at Yahoo! 資料日期 September 30, 2008 Total Nodes 4000 Total cores Data 16PB 500-node cluster 4000-node cluster write write read number of files ,000 14,000 file size (MB) , ,800 5,040,000 5,040, total MB processes tasks per node avg. throughput (MB/s) /44 read

15 Advanced Methods to deploy Cluster SSI ( Single System Image ) Multiple PCs as Single Computing Resources Image-based 15/44 homogeneous ex. SystemImager, OSCAR, Kadeploy Package-based heterogeneous easy update and modify packages ex. FAI, DRBL Other deploy tools Rocks : RPM only cfengine : configuration engine

16 Comparison of Cluster Deploy Tools Distribution Support Diskless/ Sysmless Type Node configuration tools Cluster management tools Database installation System Imager ALL Yes Image Yes No No OSCAR RPMbased Yes Image Yes Yes No Kadeploy ALL No Image Yes Yes Yes DRBL ALL Yes Package Yes Yes No FAI DebianBased Yes Package Yes No No 16/44

17 PART 2-1 : Hadoop Deployment Tool Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw 17/44

18 18/44 Source: Deploying hadoop with smartfrog

19 19/44 Source: Deploying hadoop with smartfrog

20 20/44 Source: Deploying hadoop with smartfrog

21 21/44 Source: Deploying hadoop with smartfrog

22 PART 2-2 : 企鵝龍與再生龍 DRBL and Clonezilla Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw 22/44

23 何謂企鵝龍 What is DRBL?? Diskless Remote Boot in Linux Network is cheap, Man Hour is expansive. In short, DRBL is... Use network cable to replace SATA cable All student PCs are connected to one single server Diskfull PC = + Diskless PC 23/44 + Server source:

24 何謂再生龍 What is Clonezilla?? Clone ( 複製 ) + zilla = Clonezilla ( 再生龍 ) Open Source Alternative to Norton Ghost Support Windows, Linux and Mac Disk to Disk Disk to Image Image to N Disks 24/44

25 PART 2-3 : 企鵝龍的開機原理 How does DRBL work? Jazz Wang Yao-Tsung Wang jazz@nchc.org.tw 25/44

26 1st, We install Base System of GNU/ Linux on Management Node. You can choose: Redhat, Fedora, CentOS, Mandriva, Ubuntu, Debian,... GNU Libc Kernel Module 26/44 Linux Kernel Boot Loader

27 2nd, We install DRBL package and configure it as DRBL Server. There are lots of service needed: SSHD, DHCPD, TFTPD, NFS Server, NIS Server, YP Server... Network Booting NFS Perl TFTPD DHCPD SSHD Bash DRBL Server based on existing Open Source and keep Hacking! 27/44 Account Mgnt. NIS GNU Libc Kernel Module Linux Kernel Boot Loader YP

28 After running drblsrv -i & drblpush -i, there will be pxelinux, vmlinux-pex, initrd-pxe in TFTPROOT, and different configuration files for each Compute Node in NFSROOT NFS 28/44 TFTPD DHCPD SSHD NIS Config. Files Ex. hostname GNU Libc initrd-pxe vmlinuz-pxe Kernel Module pxelinux Linux Kernel Boot Loader YP

29 3nd, We enable PXE function in BIOS configuration. BIOS PXE NFS 29/44 BIOS PXE BIOS PXE TFTPD DHCPD SSHD BIOS PXE NIS Config. Files Ex. hostname GNU Libc initrd-pxe vmlinuz-pxe Kernel Module pxelinux Linux Kernel Boot Loader YP

30 While Booting, PXE will query IP address from DHCPD. BIOS PXE NFS 30/44 BIOS PXE BIOS PXE TFTPD DHCPD SSHD BIOS PXE NIS Config. Files Ex. hostname GNU Libc initrd-pxe vmlinuz-pxe Kernel Module pxelinux Linux Kernel Boot Loader YP

31 While Booting, PXE will query IP address from DHCPD. IP 1 NFS 31/44 IP 2 IP 3 TFTPD DHCPD SSHD IP 4 NIS Config. Files Ex. hostname GNU Libc initrd-pxe vmlinuz-pxe Kernel Module pxelinux Linux Kernel Boot Loader YP

32 After PXE get its IP address, it will download booting files from TFTPD. IP 1 NFS 32/44 IP 2 IP 3 TFTPD DHCPD SSHD IP 4 NIS Config. Files Ex. hostname GNU Libc initrd-pxe vmlinuz-pxe Kernel Module pxelinux Linux Kernel Boot Loader YP

33 initrd vmlinuz initrd vmlinuz initrd vmlinuz initrd vmlinuz pxelinux IP 1 pxelinux IP 2 pxelinux IP 3 pxelinux IP 4 NFS 33/44 TFTPD DHCPD SSHD NIS Config. Files Ex. hostname GNU Libc initrd-pxe vmlinuz-pxe Kernel Module pxelinux Linux Kernel Boot Loader YP

34 initrd vmlinuz initrd vmlinuz initrd vmlinuz initrd vmlinuz pxelinux IP 1 pxelinux IP 2 pxelinux IP 3 pxelinux IP 4 NFS TFTPD DHCPD SSHD NIS GNU Libc Config. Files After downloading booting Ex. hostname YP files, scripts in initrd-pxe config Kernelwill Module initrd-pxe vmlinuz-pxe for eachlinux Kernel Node. NFSROOT Compute 34/44 pxelinux Boot Loader

35 Config. 1 initrd Config. 2 initrd Config. 3 initrd Config. 4 initrd vmlinuz pxelinux vmlinuz pxelinux vmlinuz pxelinux vmlinuz pxelinux IP 1 IP 2 IP 3 IP 4 NFS 35/44 TFTPD DHCPD SSHD NIS Config. Files Ex. hostname GNU Libc initrd-pxe vmlinuz-pxe Kernel Module pxelinux Linux Kernel Boot Loader YP

36 Perl Bash Perl Bash Perl Bash Perl Bash SSHD SSHD SSHD SSHD Applications and Services will also deployed to each Compute Node via NFS... NFS Perl 36/44 TFTPD DHCPD SSHD Bash NIS DRBL Server YP

37 SSHD SSHD SSHD SSHD With the help of NIS and YP, You can login each Compute Node with the Same ID / PASSWORD stored in DRBL Server! SSH Client NFS TFTPD DHCPD SSHD NIS DRBL Server 37/44 YP

38 PART 2-1: 當企鵝龍遇上小飛象 When DRBL meet Hadoop Jazz Wang Yao-Tsung Wang 38/44

39 Deploy Hadoop Using DRBL Under development. Need packaging. drbl-hadoop mounting local hard disk for HDFS svn co hadoop-register Website and ssh applet svn co 39/44

40 About hadoop.nchc.org.tw DRBL Server x 1 (hadoop) with more space for /home and /tftpboot DRBL Client x 19 (hadoop101~hadoop119) Using Cloudera Hadoop Debian Packages Use drbl-hadoop and cloudera's init.d script to deploy hadoop Use hadoop-register to host web service and ssh applet 40/44

41 Lesson Learn Cloudera Hadoop Package use init.d script to start/stop... Creat 500 users in advanced Use DRBL build-in command /opt/drbl/sbin/drbl-useradd Setup default HDFS home directory name node, data node, job tracker, task tracker Use for loop to run hadoop fs -mkdir tmp for each user Setup permission of user HDFS folders Use for loop to run hadoop dfs -chown $(id) /usr/$(id) HDFS use the space of /var/lib/hadoop/cache/hadoop/dfs MapReduce use the space of /var/lib/hadoop/cache/hadoop/mapred 41/44

42 結論 Conclusion Thanks to Cloudera to provide Hadoop related packages. Benefits DRBL save your time and money. It make your life easier. It's developed by Taiwan developers. Easy to communicate. Using Network Booting could save power consumption... Weakness 42/44 DRBL-Hadoop currently is only good for building experimental Hadoop Cluster. If you are looking for operational product, maybe you can try SmartFrog.

43 PART 2-2: Live Demo Jazz Wang Yao-Tsung Wang 43/44

44 Demo Network Topology * Wireless DHCP Server Physical Network Device * VMNet 8 DHCP Server Running DRBL Live CD eth0 = NAT eth1 = Host-only * VMNet 1 Host-Only Diskless Clients

45 Questions? Jazz Wang Yao-Tsung Wang 45/44