Configuration of High Performance Computing for Medical Imaging and Processing. SunGridEngine 6.2u5

Transcription

1 Configuration of High Performance Computing for Medical Imaging and Processing SunGridEngine 6.2u5 A manual guide for installing, configuring and using the cluster. Mohammad Naquiddin Abd Razak Summer

2 Table of Contents Introduction... 1 Grid Engine Installation Installation Prerequisites... 2 Installing Ubuntu Installing Java Version _ Checking your IP address... 3 Edit /etc/hosts file... 4 Installing Master Host... 6 Installing Exec/Submit Hosts... 7 Reinstall submit/exec host... 9 Sun Grid Engine Configuration Configure Hosts Adding new host group Adding Admin/Submit/Exec Host Configure Users Configure Queues Adding New Queues Attributes, Explanations and Configurations Complex Resource Configuration Adding a New Complex Resource Attribution: Resource Quota Configuration Network File Storage (NFS) Setup Mounting /share Mounting Alexandria Configure Status Monitor Using the cluster Submitting job Options for qstat Job Control Checking job status List of Nodes and Computers in MASI Lab... 26

3 Main Nodes: Added computers/nodes: Acknowledgements... 27

4 Introduction The MASI Lab has taken another step in fulfilling demands for High Performance Computing (HPC) applications in medical imaging and processing. 12 rock-dudes- CPU processors were bought where each CPU consists of 12 core processors, 273GHz systems with 47.25GB of RAM, and a GPU. These powerful machines have been connected via network file system (NFS) together with several computers available in MASI Lab. Thanks to Sun Grid Engine for providing free software which enables us to apply resource management to distribute jobs across the grid. This documentation compiles step by step procedures which consist of installation process, GridEngine configuration and NFS setup. In order to get the GridEngine working, you need to tell the master host (mastadon) about the new nodes that you want to add into the system, and also setting up a network file sharing system (NFS). This sounds simple but it is easier said than done. Together with this documentation is a simple tutorial on submitting job. You can find almost every basic thing that you need before getting your grid working in this documentation. Please go to Sun Grid Engine website ( )for more details. Don t get frustrated if things are not working yet. There are solutions for every problem 1

5 Grid Engine Installation 101 Installation Prerequisites Below are tools you need before installing master host and execution host: Ubuntu Java version 1.6.0_22 Know your IP address Edit /etc/hosts file on both master host and execution host Installing Ubuntu Please make sure that your computer is running Ubuntu Skip this section if your computer is running Ubuntu Type the following line to check you Ubuntu version: cat /etc/issue Ubuntu \n \l If you are running an older version of Ubuntu, you must upgrade to link to the cluster. To get the latest version of Ubuntu, go to: System >> Administration >> Update manager and install updates Go to the website below to get the complete instruction. Alternatively, you could type the following command on the terminal. $sudo nano /etc/update-manager/release-upgrades $sudo change lts to normal $sudo do-release-upgrade IMPORTANT!: You may need to repeat this upgrade process several time in order to update to Ubuntu To check your status look at the contents of /etc/issue. 2

6 Installing Java Version _22 Java is required for inter-host communication. Open terminal and type the following commands: $ sudo apt-get update $ sudo apt-get install openjdk-6-jre Checking your IP address Copy and paste your IP address on text editor/gedit for further use. masi@qotsa:~$ ifconfig eth0 Link encap:ethernet HWaddr 60:eb:69:81:f6:ca inet addr: Bcast: Mask: inet6 addr: fe80::62eb:69ff:fe81:f6ca/64 Scope:Link 3

7 Edit /etc/hosts file This step is very important to ensure that your hosts are able to communicate with each other. It is like your contacts account. Without it, you will never able to call your buddies. On master host, do: $sudo nano /etc/hosts mastadon.ds.vanderbilt.edu mastadon ::1 mastadon.ds.vanderbilt.edu mastadon localhost6 <exec/submit host ip address> <machine name> Note that you are adding IP addresses of your exec/submit hosts in this file. 4

8 On Execution / Submit hosts, do: $sudo nano /etc/hosts # localhost (please remove or comment this out) masi-4.ds.vanderbilt.edu masi-4 (and replace with this line) mastadon.ds.vanderbilt.edu mastadon masi-4.ds.vanderbilt.edu masi-4 Note that you are adding IP address of your master host and your own machine in this file. It is VERY IMPORTANT to complete this step on master host and exec/submit host. Failure to do so may result hosts communication error. 5

9 Installing Master Host The master host is a brain, the center to the overall cluster activities. By default, the master host is also an administration host and submit host. I found that installing master host and execution host as Ubuntu package is super easier compared to interactive installation procedure provided by the Sun Grid Engine (SGE) documentation: 1. Make sure that you have completed the prerequisites (ie; Ubuntu 11.04, Java, Editing /etc/hosts file on master and exec/submit hosts). 2. Install GridEngine on Master Host: $sudo apt-get install gridengine-client gridengine-common gridengine-exec gridengine-master gridengine-qmon General type of mail configuration: No Configuration Configure SGE automatically: Yes SGE cell name: masicluster SGE master hostname: mastadon.ds.vanderbilt.edu 3. Please make sure that you have edited the /etc/hosts file on both master and exec/submit host file. Now you are able to invoke QMON and start adding masi-4 as a submit host and execution host. Please refer to GridEngine Configuration to add new hosts. You may skip this step and add the exec/submit hosts on the system later. 6

10 Installing Exec/Submit Hosts Submit hosts allow you to submit jobs with the qsub command and monitor the job status with the qstat command. It also enables you to use Qmon; the graphical user interface (GUI). Execution Hosts are systems that have permission to execute jobs. Therefore, queue instances are attached to the execution hosts. 1. Make sure that you have completed the prerequisites (ie; Ubuntu 11.04, Java, Editing /etc/hosts file on master and exec/submit hosts). 2. Install the GridEngine package and enter yes to all. sudo apt-get install gridengine-common gridengine-exec gridengine-client Press enter (OK) Yes General type of mail configuration: No Configuration Configure SGE automatically: Yes SGE cell name: masicluster SGE master hostname: mastadon.ds.vanderbilt.edu 3. The installation is completed. Next, login back to mastadon (master host), invoke QMON and add masi-4 as execution and submit host. 7

11 4. Once your has been added as submit/exec host, you may check your computer status by typing qstat f on mastadon (master/admin/submit host only). Make sure that the master host is able to communicate with other hosts: Positive: Negative: 5. Reinstall the submit/exec hosts if you find that they are not communicating with the master host. 8

12 Reinstall submit/exec host Do not get frustrated because it may happen a lot. This problem may be due to localhost.domain type entry (it is on /etc/hosts file). I do not have any prove but my suggestion is to do repeat the installation process until it works. 1. Make sure that you have completed the prerequisites (ie; Ubuntu 11.04, Java, Editing /etc/hosts file on master and exec/submit hosts). 2. Remove the GridEngine: $Sudo apt-get --purge remove gridengine-common gridengine-exec gridengine-client 3. This is the most important steps before reinstalling the GridEngine. The left-over execd from the previous install will cause a communication failure. 4. Kill the execd operation: masi@atdi:~$ sudo kill (or any number that appears on that column) 5. Repeat step 1-4 from Installing Submit/Exec Hosts section 9

13 Sun Grid Engine Configuration The configuration can be done on terminal or QMON. Note that the full configuration can only be done by admin host. Qmon is currently installed on Mastadon and Manchu. You can go to this website ( to configure grid engine on the terminal or the using Qmon as shown below: $sudo qmon Configure Hosts Adding new host group Host Configuration >> Click on Host Group Tab >> Click Add >> >> Insert Hosts that want to be added >> Click OK >> Done to finish Adding Admin/Submit/Exec Host Host Configuration >> Click on Administration Host/Submit Host/Execution Host >> Insert new host >> Click Done to finish 10

14 Configure Users User Configuration >> Click on Userset >> Highlight arusers >> Click Modify >> Insert new User >> Click Done to finish User Configuration >> Click on User >> Highlight arusers >> Insert new User >> Click Add and Done to finish 11

15 Configure Queues An alternative way of checking your hosts communication as well as monitoring queues performances is by clicking on the Queue Control icon. There are 2 queues available on MASI Lab Cluster: cpujob, gpujob. Cpujob runs all job on each available slots, where the total number of slots is equal to the total number of processors. Also, there is 1 GPU for each MASI Lab Cluster main node. Queue Name Queue Type Total slots Max GPU Details per node per Job Cpujob CPU 12 0 Controls non-gpy type jobs Gpujob GPU 1 1 Controls GPU-type jobs only Matjob Matlab 4, 8 0 Controls matlab job 12

16 Adding New Queues Queue Control >> Click on tab Cluster Queue >> Click Add >> Enter a queue name >> Enter New Host/Host group Below are the attributions which have been set up for cpujob and gpujob: Queue Seq_no Priority Load thresholds Slots Complexes Name cpujob 2-20 np_load_avg=1.75, 1, [@cpu4=4], None [@cpu4=np_load_avg=1], [@cpu8=np_load_avg=1] [@exechost=12], [cpu8=8] gpujob 0 20 np_load_avg= gpu=1 13

17 Attributes, Explanations and Configurations seq_no Sequencing queues into orders starting from 0. Queue Control >> Highlight a queue >> Modify >> change sequence number Priority -20 is the lowest possible queue priority and 20 is the highest possible queue priority Queue Control >> Highlight a queue >> Modify >> Change priority value Load Thresholds (np_load_avg) overload prevents the queue from receiving further jobs Queue Control >> Highlight a queue >> Modify >> Highlight a host on Attributes for Host or add a New Host on the bottom left corner >> change the np_load_avg value Slots Slots value was set to be the same as number of processors available. For gpujob, only one slot available on each node as we want only one job running on each GPU (there are one GPU for each node). Queue Control >> Highlight a queue >> Modify >> Change the slots value Complexes (gpu=1) Gpu is a new complex attribution added on cluster. This attribute was fixed to 1. However, if more gpu were added on each node, this value can simple be changed to any value. Queue Control >> Highlight Queue >> Modify >> Complexes >> Add Cosumable/Fixed Attributes (ie; gpu=1). 14

18 It is IMPORTANT for any gpu usage job to tell the cluster that they are using one gpu per job: please add this line on your script for gpu job: #$ -l gpu=1 Configuring queue on the terminal: Complex Resource Configuration Adding a New Complex Resource Attribution: Qmon> Complex Configuration> Fill in the Attributes > Add> Commit Example: A new complex resource configuration has been added to limit each node to a single GPU. 15

19 If a job with GPU usage is requested, the following command line should be added to the job script in order to request the type of queue and setting the number of GPU is equal to 1. You can set the Consumable attribute to YES if there were more GPU added on each node. #$-q gpujob #$ -l gpu=1 (we are using the new attribution here on this line) For more details about complex resource attributes configuration, click on the link below: exresourceattributes-examplesofsettingupconsumableresources 16

20 Resource Quota Configuration A resource quota configuration is used to prevent users from consuming all available resources. For the MASI Lab Cluster purpose of usage, we host group to use the total number of slots to be equal to total number of processors. By providing this limit, only 144 jobs will be running at a time (as there are 144 number of processors), each of the job occupying the empty slots available, and the rest will be queued. More information about static and dynamic resource quotas can be viewed on the link provided as below: 17

21 Network File Storage (NFS) Setup NAS is a storage device that attaches directly to a network, allowing clients to access the storage as if it were directly attached to their system. In MASI Lab Cluster, storage becomes accessible directly across the network through /share directory. Mounting /share Install packages for NFS: abdrazm@masi-7:~$ sudo apt-get install nfs-kernel-server nfs common abdrazm@masi-7:~$ sudo apt-get install smbfs Create a new folder to mount the /share file on NAS device. abdrazm@masi-7:~$ sudo mkdir /share Edit name service switch file: abdrazm@masi-7:~$ sudo nano /etc/nsswitch.conf hosts: files mdns4_minimal [NOTFOUND=return] wins dns mdns4 We need to create a credential file to mount on startup, and save the file: abdrazm@masi-7:~$ sudo nano /root/.smbcredentials username= XXXXXXXX password= XXXXXXXX Using chmod command to set file permission for /root/.smbcredentials so that only root can read and edit it: abdrazm@masi-7:~$ sudo chmod 700 /root/.smbcredentials Now we want to back up out fstab: abdrazm@masi-7:~$ sudo cp /etc/fstab /etc/fstab_old 18

22 Edit fstab: sudo nano /etc/fstab // /share /share cifs credentials=/root/.smbcredentials,iocharset=utf8,file_mode=0777,d ir_mode= Copy fstab for share-> masi-7 (/etc/fstab) root@masi-7:~# cp r /home/likewise-open/vanderbilt/abdrazm/etc/fstab /share Make a backup for /home/likewise-open directory: root@masi-7:~# mkdir /home-local root@masi-7:~# mv /home/likewise-open /home-local/ Soft link: root@masi-7:~# ln s /share/home/likewise-open /home/likewise-open Do an ls/home/ and you should see a likewise directory: root@masi-7:~# ls /home/ Log back in as your vunet id after reboot: root@masi-7:~# reboot 19

23 Mounting Alexandria sudo i abdrazm@masi-7~# apt-get install nfs-kernel-server nfs common abdrazm@masi-7~# cd / Create /home-nfs/likewise-open directory: root@masi-7:~# mkdir /home-nfs root@masi-7:~# mkdir /home-nfs/likewise-open Edit /etc/fstab. root@masi-7:~# nano /etc/fstab alexandria:/ /home-nfs/likewise-open nfs4 _netdev,auto 0 0 Mount /home-nfs/likewise-open root@masi-7:~#mount /home-nfs/likewise-open Create /homr.orig as a backup and move /home/likewise-open to /home.orig root@masi-7:~# mkdir /home.orig root@masi-7:~# mv /home/likewise-open /home.orig Soft link: root@masi-7:~# ln s /home-nfs/likewise-open /home/likewise-open Type sudo mount to check status of the mounted directory. 20

24 Configure Status Monitor This enables us to monitor machine status of each computer from Inside Masi ( sudo i ssh-keygen t rsa cat.ssh/id_rsa.pub ssh [email protected] 'cat >>.ssh/authorized_keys' The password for status is XXXXXXXX Install sshfs and autofs sudo apt-get install sshfs sudo apt-get install autofs sudo crontab e */2 * * * * /mnt/common/status/run-masi.sh 2> /dev/null > /dev/null Create /mnt/common and add to /etc/auto.master mkdir /mnt/common /mnt/common /etc/auto.sshfs Create /etc/auto.sshfs sudo nano /etc/auto.sshfs * - fstype=fuse,rw,nodev,nonempty,noatime,allow_other :sshfs\#[email protected]\:/mnt/c ommon/statu Restart the autofs server /etc/init.d/autofs restart 21

25 Using the cluster Submitting job To submit a serial job first we need to make an SGE script file: 1. Create a file on linux: 2. Job example 1: 3. Job example 2: The following commands are important to request GPU usage for a gpujob: #$-q gpujob The above command requesting gpujob script to be run into gpujob queue. #$ -l gpu=1 this tells SGE that we are requesting a complex source gpu and it is equal to 1 (recall that there is only one graphic card per node) 4. Submit the job using $qsub: 22

26 to submit an array of jobs, use the command below: 5. Check your job status using $qstat or $qstat -f 6. You can read your job script by using the cat command Options for qstat qstat qstat -f qstat u user qstat j job_id Qstat explain c -j job_id Print general status each submitted job Print full job information. Very nice and clear Print all jobs of a given user Print full job information of the given job id Print detail explanation of the given job id plus explanation on errors/pending Job Control SGE state letter symbol codes meanings: Pending Pending qw Pending, user/system hold hqw Pending, user/system hold, re-queue hrqw Running Running r Transferring t Running, re-submit Rr Transferring, re-submit Rt Suspended Job suspended s, ts Queue suspended S, ts Queue suspended by alarm T, tt All suspended with re-submit Rs, Rts, RS, RtS, RT, RtT Error All pending states with error Eqw, Ehqw, EhRqw Delete All running and suspended states with deletion dr, dt, drt, drr, ds, ds, dt, drs, drs, drt 23

27 Checking job status $ qstat : this will give you the general status of each submitted job. No information will be given if none of the job available on each queues (finished job/ none job submitted). $ qstat f : use this command to find more details about the submitted job on each queue 24

28 $ qstat explain c j jobid : Job-ID 1069 is pending as you could see on the diagram. Any job with status other than r indicates that they are not happy. We can check the reason by using qstat explain command. Please refer to table Table 4 for job status explanations. 25

29 List of Nodes and Computers in MASI Lab Main Nodes: MASI main Host IP address mastadon.ds.vanderbilt.edu Admin, Submit, Executive Isis.ds.vanderbilt.edu Executive Kyuss.ds.vanderbilt.edu Executive Godspeed.ds.vanderbilt.edu Executive Melvins.ds.vanderbilt.edu Executive Outkast.ds.vanderbilt.edu Executive Qotsa.ds.vanderbilt.edu Executive Atdi.ds.vanderbilt.edu Executive Felakuti.ds.vanderbilt.edu Executive Slayer.ds.vanderbilt.edu Executive Mogwai.ds.vanderbilt.edu Executive Pelican.ds.vanderbilt.edu Executive Added computers/nodes: MASI extra nodes Host Group Host IP address No. of CPU Submit, Executive Masi-1.ds.vanderbilt.edu Masi-2.ds.vanderbilt.edu Masi-2.ds.vanderbilt.edu Masi-4.ds.vanderbilt.edu Executive Executive Executive Executive Executive Executive

30 Acknowledgements Special thanks to.. Professor Bennett Landman He is a great person, very supportive, warm and intellectual. Albert Hampton ([email protected]), a hardworking technician, has almost every solution to fix computer problems. Elliot Hall, a great friend and a coffee lover. May God make things easier for your wedding. Andrew Asman, Xue Yang, Zhoubing Xu, Michael Esparza, Eesha Singh and Lauzon Carolyn who made the summer wonderful with smile, laugh and joy. 27