Ra - Batch Scripts. Timothy H. Kaiser, Ph.D. tkaiser@mines.edu

Transcription

1 Ra - Batch Scripts Timothy H. Kaiser, Ph.D. tkaiser@mines.edu

2 Jobs on Ra are Run via a Batch System Ra is a shared resource Purpose: Give fair access to all users Have control over where jobs are run Set limits on times and processor counts Allow exclusive access to nodes while you own them

3 Batch Systems Create a script that describes: What you want to do Resources Time required What resources (# nodes) you want How long you want the resources Submit you script to the queuing system Wait for some time for your job to start Your job runs and you get your output

4 Moab -Top level Batch System Product of Cluster Resources products/moab-cluster-suite/workloadmanager.php Marketing pitch from the web page: Moab Workload Manager is a policy-based job scheduler and event engine that enables utility-based computing for clusters. Moab Workload Manager combines intelligent scheduling of resources with advanced reservations to process jobs on the right resources at the right time. It also provides flexible policy and event engines that process workloads faster and in line with set business requirements and priorities.

5 Moab Set up queues policy Hands actual job submission to Torque Torque - a.k.a. pbs Torque is a descendent of the open source version of PBS Moab has the commands of pbs but adds some. The most important is msub

6 A Simple Script for a MPI job!/bin/bash #PBS -l nodes=1:ppn=8 #PBS -l walltime=02:00:00 #PBS -N testio #PBS -o stdout #PBS -e stderr #PBS -V #PBS -m abe #PBS -M tkaiser@mines.edu # cd $PBS_O_WORKDIR mpiexec -n 8 c_ex00 Scripts contain normal shell commands and comments designated with a # that are interpreted by PBS

7 Running and monitoring jobs ~/mydir]$ msub mpmd Use msub to submit the job qstat to see its status (Q,R,C) qdel to kill the job if needed 8452 ~/mydir]$ qstat 8452 Job id Name User Time Use S Queue ra testio tkaiser 0 R SHORT [tkaiser@ra ~/mydir]$ [tkaiser@ra ~/mydir]$ [tkaiser@ra ~/mydir]$ qstat -u tkaiser Job id Name User Time Use S Queue ra testio tkaiser 0 R SHORT [tkaiser@ra ~/mydir]$ [tkaiser@ra ~/mydir]$ [tkaiser@ra ~/mydir]$ qdel 8452

8 A Simple Script for a MPI job Command!/bin/bash #PBS -l nodes=1:ppn=8 #PBS -l walltime=02:00:00 #PBS -N testio #PBS -o stdout #PBS -e stderr #PBS -V #PBS -m abe #PBS -M tkaiser@mines.edu Comment We will run this job using the "bash" shell We want 1 node with 8 processors for a total of 8 processors We will run for up to 2 hours The name of our job is testio Standard output from our program will go to a file stdout Error output from our program will go to a file stderr Very important! Exports all environment variables from the submitting shell into the batch shell. Send mail on abort, begin, end use n to suppress Where to send to (cell phone?) # Not important. Just a separator line. cd $PBS_O_WORKDIR mpiexec -n 8 c_ex00 Very important! Go to directory $PBS_O_WORKDIR which is the directory which is where our script resides Run the MPI program c_ex00 on 8 computing cores.

9 #PBS -l More Script notes Used to specify resource requests Can have multiple per -l lines per script #PBS -l nodes=2:ppn=8:fat Ra has 16 Gbyte and 32 Gbyte nodes Specify the 32 Gbyte nodes #PBS -l naccesspolicy=singlejob Grant exclusive access to a node Not normally needed

10 PBS Variables Variable PBS_JOBID PBS_O_WORKDIR PBS_NNODES PBS_NODEFILE PBS_O_HOME PBS_JOBNAME PBS_QUEUE PBS_MOMPORT PBS_O_HOST PBS_O_LANG PBS_O_LOGNAME PBS_O_PATH PBS_O_SHELL PBS_JOBCOOKIE PBS_TASKNUM PBS_NODENUM Meaning unique PBS job ID jobs submission directory number of nodes requested file with list of allocated nodes home dir of submitting user user specified job name job queue active port for mom daemon host of currently running job language variable for job name of submitting user path to executables used in job script script shell job cookie number of tasks requested (see pbsdsh) node offset number (see pbsdsh)

11 Where s my Output? Standard Error and Standard Out don t show up until after your job is completed Output is temporarily stored in /tmp on your nodes You can ssh to your nodes and see your output Looking into changing this behavior, or at least make it optional You can pipe output to a file: mpiexec myjob > myjob.$pbs_jobid

12 Useful Moab Commands Command canceljob checkjob mdiag mjobctl mrsvctl mshow msub releasehold releaseres sethold showq showres showstart showstate cancel job Description provide detailed status report for specified job provide diagnostic reports for resources, workload, and scheduling control and modify job create, control and modify reservations displays various diagnostic messages about the system and job queues submit a job (Donʼt use qsub) release job defers and holds release reservations set job holds show queued jobs show existing reservations show estimates of when job can/will start show current state of resources

13 Useful Torque commands Command tracejob pbsnodes qalter qdel qhold qrls qrun qsig qstat pbsdsh qsub Description trace job actions and states recorded in TORQUE logs view/modify batch status of compute nodes modify queued batch jobs delete/cancel batch jobs hold batch jobs release batch job holds start a batch job send a signal to a batch job view queues and jobs launch tasks within a parallel job submit a job (Use msub instead)

14 Running Interactive

15 Interactive Runs It is possible to grab a collection of nodes and run parallel interactively Nodes are yours until you quit Might be useful during development Running a number of small parallel jobs Required (almost) for running parallel debuggers

16 Multistep Process Ask for the nodes using msub msub -q INTERACTIVE -I -V -l nodes=1 Go to your working directory Run your job(s) Logout to release your nodes You are automatically Logged on to the nodes you are given Ignore: cp: cannot stat `/opt/torque/mom_priv/jobs/73156.ra.local.sc': No such file or directory Don t abuse it, lest you be shot.

17 Example ~/guide]$msub -q INTERACTIVE -I -V -l nodes=1 -l naccesspolicy=singlejob msub: waiting for job 1280.ra.mines.edu to start msub: job 1280.ra.mines.edu ready ~]$cd guide ~/guide]$mpiexec -n 4 c_ex00 Hello from 0 of 4 on compute-9-8.local Hello from 1 of 4 on compute-9-8.local Hello from 2 of 4 on compute-9-8.local Hello from 3 of 4 on compute-9-8.local [tkaiser@compute-9-8 ~/guide]$ [tkaiser@compute-9-8 ~/guide]$mpiexec -n 16 c_ex00 Hello from 1 of 16 on compute-9-8.local Hello from 0 of 16 on compute-9-8.local Hello from 3 of 16 on compute-9-8.local Hello from 4 of 16 on compute-9-8.local Hello from 15 of 16 on compute-9-8.local Hello from 2 of 16 on compute-9-8.local Hello from 6 of 16 on compute-9-8.local [tkaiser@compute-9-8 ~/guide]$ [tkaiser@compute-9-8 ~/guide]$exit logout Ensures exclusive access msub: job 1280.ra.mines.edu completed [tkaiser@ra ~/guide]$