Using Parallel Computing to Run Multiple Jobs
|
|
- Horatio Green
- 8 years ago
- Views:
Transcription
1 Beowulf Training Using Parallel Computing to Run Multiple Jobs Jeff Linderoth August 5, 2003 August 5, 2003 Beowulf Training Running Multiple Jobs Slide 1
2 Outline Introduction to Scheduling Software The Wonderful World of PBS The Equally Wonderful World of Condor Lab Time. Why do we need scheduling software? August 5, 2003 Beowulf Training Running Multiple Jobs Slide 2
3 Resource Scheduling So people don't ght over the resources! Schedulers... Locate appropriate resources, Manage resources, so multiple processes don't conict over the same processor Ensure a fairness policy, Are integrated with accounting software. The schedulers on the Beowulf cluster are PBS and Condor. August 5, 2003 Beowulf Training Running Multiple Jobs Slide 3
4 Mmmmmmmmmmmmmm. Pie Our rst computational task will be to estimate π by numerical integration. Everyone knows x 2 dx = arctan(x) 1 x=0 = arctan(1) = π 4. August 5, 2003 Beowulf Training Running Multiple Jobs Slide 4
5 The Rectangle Rule /(1+x*x) August 5, 2003 Beowulf Training Running Multiple Jobs Slide 5
6 A Program to Estimate π I've written a π-calculator for you. cd mkdir compute-pi cd compute-pi cp /tmp/training/session2/pi1.c. gcc pi1.c -lm -o pi1./pi This is not a parallel program. Just a simple (one process) program. Nevertheless, we must submit it through a scheduling system to run it on the Beowulf cluster. August 5, 2003 Beowulf Training Running Multiple Jobs Slide 6
7 Running with PBS A simple four step process... Create a PBS submission script Submit the script to the PBS system using the command qsub PBS runs the script on the rst available resources PBS collects output for user's inspection August 5, 2003 Beowulf Training Running Multiple Jobs Slide 7
8 The PBS Submission Script Overview (1) You make a request for resources, (2) PBS will allocate a node pool to fulll your request. (3) Now you have to tell the node pool what to do! Both steps (1) and (3) are accomplished through the PBS submission script The script contains PBS request statements Shell commands that will run your job on the allocated resources. The shell commands are executed on the rst node in your allocated nodes August 5, 2003 Beowulf Training Running Multiple Jobs Slide 8
9 Our First PBS Submission Script #PBS -q small #PBS -l nodes=1:public #PBS -l cput=00:05:00 #PBS -V echo "The PBS job ID is: ${PBS_JOBID}" echo "The PBS Node File is" cat $PBS_NODEFILE $HOME/compute-pi/pi1 100 August 5, 2003 Beowulf Training Running Multiple Jobs Slide 9
10 Format of the PBS Submission Script Lines that begin with #PBS are PBS directives Everything else is a shell command Shell commands are just things that you would type at the regular login-prompt. But you can also do fancy looping and conditions. chapter/bashref toc.html After the PBS commands, you put any commands you would like. Usually the command to run your program is usually a good one to include. :-) Again, this is executed on the rst node. August 5, 2003 Beowulf Training Running Multiple Jobs Slide 10
11 Breaking It Down. PBS Directives -q Species the queue in which to place the job. We have two queues, small and large small Max CPU time 20 minutes/process. large Lower priority than jobs in small queue -l Denes the resources that are required by the job and establishes a limit to the amount of resource that can be consumed. -V Declares that all environment variables in the qsub command's environment are to be exported to the batch job. If you would like the PBS job to inherit the same environment as the one you are currently running in (same PATH variable, etc), you should include this directive. August 5, 2003 Beowulf Training Running Multiple Jobs Slide 11
12 The -l Story For resources, you will typically only need to declare the number of nodes, which class of nodes you request #PBS -l nodes=4:public the maximum cpu time #PBS -l cput=00:15:00 For the truly brave and curious the command is man pbs resources August 5, 2003 Beowulf Training Running Multiple Jobs Slide 12
13 PBS The Big Three qsub Submit a PBS job qstat Check the status of a PBS job qdel Delete a PBS job man <command> will give you lots more information August 5, 2003 Beowulf Training Running Multiple Jobs Slide 13
14 Let's do it! compute-pi-1]$ qsub run.pbs 5972.fire1 compute-pi-1]$ qstat -a fire1: Req d Req d Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time fire1 jtl3 small run.pbs :20 E -- Note that the job ID is printed for you when you submit the job qstat -a : Shows the status of all jobs August 5, 2003 Beowulf Training Running Multiple Jobs Slide 14
15 Looking at the Output By default standard output goes to <scriptname>.o<job number> By default standard error goes to <scriptname>.e<job number> compute-pi-1]$ cat run.pbs.o5972 The PBS job ID is: 5972.fire1 The PBS Node File is fire34 pi is about Error is e-02 Note how the PBS environment variables are interpreted in the script. August 5, 2003 Beowulf Training Running Multiple Jobs Slide 15
16 Other Cool PBS Stuff You May Want To Do #PBS -N <Name> : Name your job #PBS -o <File.out> : Redirect standard output to File.out #PBS -e <File.err> : Redirect standard error to File.err #PBS -m -M : Mail options Job dependencies For a list of all PBS command le options... man qsub Any PBS Questions? August 5, 2003 Beowulf Training Running Multiple Jobs Slide 16
17 Condor For purposes of this discussion, think of Condor as a different scheduler. Condor is a bit more fancy. Used often for nondedicated resources. (Will run only when no one else would use the machine). Checkpointing/Migration Remote I/O Likely, the accounting charge will be less for jobs submit to the Condor scheduler. August 5, 2003 Beowulf Training Running Multiple Jobs Slide 17
18 Checkpointing/Migration Professor s Machine Professor Arrives } 5 min 5am 8am Grad Student s Machine Checkpoint Server Grad Student Arrives Grad Student Leaves 8:10am } 12pm 5 min August 5, 2003 Beowulf Training Running Multiple Jobs Slide 18
19 Condor Universes Condor jobs are submit to a specic Condor Universe Standard Has cool features like checkpointing and migration of jobs Requires special linking of your program Vanilla No cool condor features (regular) MPI/PVM Not mentioned here today, but they exist. August 5, 2003 Beowulf Training Running Multiple Jobs Slide 19
20 Compiling for Condor Standard Universe Put the command condor compile in front of your normal link line. condor]$ condor compile gcc pi1.c -o pi1-standard -lm Vanilla Universe Do nothing Now Condor submission is like PBS submission Different command (job description) le Different submission/montoring commands August 5, 2003 Beowulf Training Running Multiple Jobs Slide 20
21 A Sample Condor Submission File universe = standard executable = pi1-standard arguments = output = pi1.out error = pi1.err notification = Complete notify_user = jtl3@lehigh.edu getenv = True rank = kflops queue man condor submit August 5, 2003 Beowulf Training Running Multiple Jobs Slide 21
22 The Big Four condor submit <job.condor> Submit a job to the Condor scheduler condor q Check the status of the queue of Condor jobs condor status Check the status of the condor pool condor rm <jobid> Delete a Condor job August 5, 2003 Beowulf Training Running Multiple Jobs Slide 22
23 Let's Do It! condor]$ condor_submit run.condor Submitting job(s). 1 job(s) submitted to cluster 16. [jtl3@fire1 condor]$ condor_q -- Submitter: fire1.cluster : < :32777> : fire1.cluster ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 16.0 jtl3 8/4 11: :00:16 R pi1-standard [jtl3@fire1 condor]$ cat pi1.out pi is about Error is e-09 I could do condor rm 16.0 Any Condor questions? August 5, 2003 Beowulf Training Running Multiple Jobs Slide 23
24 Quit Wasting My Time! OK, Linderoth, I thought today was supposed to be about parallel computing! That will be the focus of the next section(s) For now, let's do some simple parallel computing. Suppose I'd like to run the same executable pi1, but with many different input les or parameters. Use the multiple processors to get your work done faster August 5, 2003 Beowulf Training Running Multiple Jobs Slide 24
25 Running Many Jobs We need a way to easily submit many different jobs We will use the shell's scripting capabilities PBS Use a template command le and the sed utility Condor Use the -a ag to condor submit August 5, 2003 Beowulf Training Running Multiple Jobs Slide 25
26 PBS Run Multiple Jobs. Step #1 Create a template submission le. #!/bin/bash #PBS -q small #PBS -l nodes=1:public #PBS -l walltime=00:05:00 #PBS -V echo "The PBS job ID is: ${PBS_JOBID}" echo "The PBS Node File is" cat $PBS_NODEFILE /home/jtl3/class/pbs/pi1 XXX N XXX August 5, 2003 Beowulf Training Running Multiple Jobs Slide 26
27 PBS Run Multiple Jobs. Step #2 Create a shell script to do the multiple submission #!/bin/bash for n in do sed s/xxx_n_xxx/$n/g run.pbs.template > run.pbs.tmp qsub run.pbs.tmp rm run.pbs.tmp done The sed commands replaces all occurances of the pattern XXX N XXX with the variable $n in run.pbs.template. August 5, 2003 Beowulf Training Running Multiple Jobs Slide 27
28 PBS Run Multiple Jobs. pbs]$ sh run-many.sh 5989.fire fire fire fire fire1 sh the script you created Any questions about PBS multiple job submission? August 5, 2003 Beowulf Training Running Multiple Jobs Slide 28
29 Condor Run Multiple Jobs Example condor submit allows the user to override statements in the submission le. Use the -a ag This makes our scripting life easier we don't need to use sed August 5, 2003 Beowulf Training Running Multiple Jobs Slide 29
30 Condor Run Multiple Jobs. Step #1 Create the Condor submission le Note no arguments or output lines! executable = pi1-standard universe = standard notification = Complete notify_user = jtl3@lehigh.edu getenv = True rank = kflops queue August 5, 2003 Beowulf Training Running Multiple Jobs Slide 30
31 The Condor Multiple Job Submission Script Create the condor multiple job submission script Note the use of the -a option! #!/bin/bash for n in do condor_submit -a "arguments = $n" -a "output = pi.$n.out"\ run.condor.many done August 5, 2003 Beowulf Training Running Multiple Jobs Slide 31
32 Multiple Condor Submission Example condor]$ sh run-many.sh Submitting job(s). 1 job(s) submitted to cluster 32. Submitting job(s). 1 job(s) submitted to cluster 33. Submitting job(s). 1 job(s) submitted to cluster 34. Submitting job(s). 1 job(s) submitted to cluster 35. Submitting job(s). 1 job(s) submitted to cluster 36. [jtl3@fire1 condor]$ condor_q -- Submitter: fire1.cluster : < :32777> : fire1.cluster ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 33.0 jtl3 8/4 12: :00:01 R pi1-standard jtl3 8/4 12: :00:00 R pi1-standard jtl3 8/4 12: :00:00 I pi1-standard jtl3 8/4 12: :00:00 I pi1-standard jobs; 2 idle, 2 running, 0 held August 5, 2003 Beowulf Training Running Multiple Jobs Slide 32
33 The End! Schedulers are required for use in a parallel computing environment PBS and Condor are cool You can do parallel computing even with MPI The Beowulf cluster can by a CPU cycle server for your research! August 5, 2003 Beowulf Training Running Multiple Jobs Slide 33
Miami University RedHawk Cluster Working with batch jobs on the Cluster
Miami University RedHawk Cluster Working with batch jobs on the Cluster The RedHawk cluster is a general purpose research computing resource available to support the research community at Miami University.
More informationPBS Tutorial. Fangrui Ma Universit of Nebraska-Lincoln. October 26th, 2007
PBS Tutorial Fangrui Ma Universit of Nebraska-Lincoln October 26th, 2007 Abstract In this tutorial we gave a brief introduction to using PBS Pro. We gave examples on how to write control script, and submit
More informationBatch Scheduling and Resource Management
Batch Scheduling and Resource Management Luke Tierney Department of Statistics & Actuarial Science University of Iowa October 18, 2007 Luke Tierney (U. of Iowa) Batch Scheduling and Resource Management
More informationGrid 101. Grid 101. Josh Hegie. grid@unr.edu http://hpc.unr.edu
Grid 101 Josh Hegie grid@unr.edu http://hpc.unr.edu Accessing the Grid Outline 1 Accessing the Grid 2 Working on the Grid 3 Submitting Jobs with SGE 4 Compiling 5 MPI 6 Questions? Accessing the Grid Logging
More informationIntroduction to Running Hadoop on the High Performance Clusters at the Center for Computational Research
Introduction to Running Hadoop on the High Performance Clusters at the Center for Computational Research Cynthia Cornelius Center for Computational Research University at Buffalo, SUNY 701 Ellicott St
More informationQuick Tutorial for Portable Batch System (PBS)
Quick Tutorial for Portable Batch System (PBS) The Portable Batch System (PBS) system is designed to manage the distribution of batch jobs and interactive sessions across the available nodes in the cluster.
More informationTutorial: Using WestGrid. Drew Leske Compute Canada/WestGrid Site Lead University of Victoria
Tutorial: Using WestGrid Drew Leske Compute Canada/WestGrid Site Lead University of Victoria Fall 2013 Seminar Series Date Speaker Topic 23 September Lindsay Sill Introduction to WestGrid 9 October Drew
More informationLinux für bwgrid. Sabine Richling, Heinz Kredel. Universitätsrechenzentrum Heidelberg Rechenzentrum Universität Mannheim. 27.
Linux für bwgrid Sabine Richling, Heinz Kredel Universitätsrechenzentrum Heidelberg Rechenzentrum Universität Mannheim 27. June 2011 Richling/Kredel (URZ/RUM) Linux für bwgrid FS 2011 1 / 33 Introduction
More informationGrid Engine Users Guide. 2011.11p1 Edition
Grid Engine Users Guide 2011.11p1 Edition Grid Engine Users Guide : 2011.11p1 Edition Published Nov 01 2012 Copyright 2012 University of California and Scalable Systems This document is subject to the
More informationNEC HPC-Linux-Cluster
NEC HPC-Linux-Cluster Hardware configuration: 4 Front-end servers: each with SandyBridge-EP processors: 16 cores per node 128 GB memory 134 compute nodes: 112 nodes with SandyBridge-EP processors (16 cores
More informationJob Scheduling with Moab Cluster Suite
Job Scheduling with Moab Cluster Suite IBM High Performance Computing February 2010 Y. Joanna Wong, Ph.D. yjw@us.ibm.com 2/22/2010 Workload Manager Torque Source: Adaptive Computing 2 Some terminology..
More informationIntroduction to Sun Grid Engine (SGE)
Introduction to Sun Grid Engine (SGE) What is SGE? Sun Grid Engine (SGE) is an open source community effort to facilitate the adoption of distributed computing solutions. Sponsored by Sun Microsystems
More informationInstalling and running COMSOL on a Linux cluster
Installing and running COMSOL on a Linux cluster Introduction This quick guide explains how to install and operate COMSOL Multiphysics 5.0 on a Linux cluster. It is a complement to the COMSOL Installation
More information1.0. User Manual For HPC Cluster at GIKI. Volume. Ghulam Ishaq Khan Institute of Engineering Sciences & Technology
Volume 1.0 FACULTY OF CUMPUTER SCIENCE & ENGINEERING Ghulam Ishaq Khan Institute of Engineering Sciences & Technology User Manual For HPC Cluster at GIKI Designed and prepared by Faculty of Computer Science
More informationGrid Engine Basics. Table of Contents. Grid Engine Basics Version 1. (Formerly: Sun Grid Engine)
Grid Engine Basics (Formerly: Sun Grid Engine) Table of Contents Table of Contents Document Text Style Associations Prerequisites Terminology What is the Grid Engine (SGE)? Loading the SGE Module on Turing
More informationCluster Computing With R
Cluster Computing With R Stowers Institute for Medical Research R/Bioconductor Discussion Group Earl F. Glynn Scientific Programmer 18 December 2007 1 Cluster Computing With R Accessing Linux Boxes from
More informationBeyond Windows: Using the Linux Servers and the Grid
Beyond Windows: Using the Linux Servers and the Grid Topics Linux Overview How to Login & Remote Access Passwords Staying Up-To-Date Network Drives Server List The Grid Useful Commands Linux Overview Linux
More informationCluster@WU User s Manual
Cluster@WU User s Manual Stefan Theußl Martin Pacala September 29, 2014 1 Introduction and scope At the WU Wirtschaftsuniversität Wien the Research Institute for Computational Methods (Forschungsinstitut
More informationRunning applications on the Cray XC30 4/12/2015
Running applications on the Cray XC30 4/12/2015 1 Running on compute nodes By default, users do not log in and run applications on the compute nodes directly. Instead they launch jobs on compute nodes
More informationNYUAD HPC Center Running Jobs
NYUAD HPC Center Running Jobs 1 Overview... Error! Bookmark not defined. 1.1 General List... Error! Bookmark not defined. 1.2 Compilers... Error! Bookmark not defined. 2 Loading Software... Error! Bookmark
More informationUsing WestGrid. Patrick Mann, Manager, Technical Operations Jan.15, 2014
Using WestGrid Patrick Mann, Manager, Technical Operations Jan.15, 2014 Winter 2014 Seminar Series Date Speaker Topic 5 February Gino DiLabio Molecular Modelling Using HPC and Gaussian 26 February Jonathan
More informationUsing the Yale HPC Clusters
Using the Yale HPC Clusters Stephen Weston Robert Bjornson Yale Center for Research Computing Yale University Oct 2015 To get help Send an email to: hpc@yale.edu Read documentation at: http://research.computing.yale.edu/hpc-support
More informationMartinos Center Compute Clusters
Intro What are the compute clusters How to gain access Housekeeping Usage Log In Submitting Jobs Queues Request CPUs/vmem Email Status I/O Interactive Dependencies Daisy Chain Wrapper Script In Progress
More informationHigh-Performance Reservoir Risk Assessment (Jacta Cluster)
High-Performance Reservoir Risk Assessment (Jacta Cluster) SKUA-GOCAD 2013.1 Paradigm 2011.3 With Epos 4.1 Data Management Configuration Guide 2008 2013 Paradigm Ltd. or its affiliates and subsidiaries.
More informationSLURM: Resource Management and Job Scheduling Software. Advanced Computing Center for Research and Education www.accre.vanderbilt.
SLURM: Resource Management and Job Scheduling Software Advanced Computing Center for Research and Education www.accre.vanderbilt.edu Simple Linux Utility for Resource Management But it s also a job scheduler!
More informationHodor and Bran - Job Scheduling and PBS Scripts
Hodor and Bran - Job Scheduling and PBS Scripts UND Computational Research Center Now that you have your program compiled and your input file ready for processing, it s time to run your job on the cluster.
More informationStreamline Computing Linux Cluster User Training. ( Nottingham University)
1 Streamline Computing Linux Cluster User Training ( Nottingham University) 3 User Training Agenda System Overview System Access Description of Cluster Environment Code Development Job Schedulers Running
More informationParallel Debugging with DDT
Parallel Debugging with DDT Nate Woody 3/10/2009 www.cac.cornell.edu 1 Debugging Debugging is a methodical process of finding and reducing the number of bugs, or defects, in a computer program or a piece
More informationBatch Scripts for RA & Mio
Batch Scripts for RA & Mio Timothy H. Kaiser, Ph.D. tkaiser@mines.edu 1 Jobs are Run via a Batch System Ra and Mio are shared resources Purpose: Give fair access to all users Have control over where jobs
More informationAn Introduction to High Performance Computing in the Department
An Introduction to High Performance Computing in the Department Ashley Ford & Chris Jewell Department of Statistics University of Warwick October 30, 2012 1 Some Background 2 How is Buster used? 3 Software
More informationCloud Computing. Lectures 3 and 4 Grid Schedulers: Condor 2014-2015
Cloud Computing Lectures 3 and 4 Grid Schedulers: Condor 2014-2015 Up until now Introduction. Definition of Cloud Computing. Grid Computing: Schedulers: Condor architecture. Summary Condor: user perspective.
More informationManual for using Super Computing Resources
Manual for using Super Computing Resources Super Computing Research and Education Centre at Research Centre for Modeling and Simulation National University of Science and Technology H-12 Campus, Islamabad
More informationGetting Started with HPC
Getting Started with HPC An Introduction to the Minerva High Performance Computing Resource 17 Sep 2013 Outline of Topics Introduction HPC Accounts Logging onto the HPC Clusters Common Linux Commands Storage
More informationWork Environment. David Tur HPC Expert. HPC Users Training September, 18th 2015
Work Environment David Tur HPC Expert HPC Users Training September, 18th 2015 1. Atlas Cluster: Accessing and using resources 2. Software Overview 3. Job Scheduler 1. Accessing Resources DIPC technicians
More informationSGE Roll: Users Guide. Version @VERSION@ Edition
SGE Roll: Users Guide Version @VERSION@ Edition SGE Roll: Users Guide : Version @VERSION@ Edition Published Aug 2006 Copyright 2006 UC Regents, Scalable Systems Table of Contents Preface...i 1. Requirements...1
More informationIntroduction to the SGE/OGS batch-queuing system
Grid Computing Competence Center Introduction to the SGE/OGS batch-queuing system Riccardo Murri Grid Computing Competence Center, Organisch-Chemisches Institut, University of Zurich Oct. 6, 2011 The basic
More informationSLURM: Resource Management and Job Scheduling Software. Advanced Computing Center for Research and Education www.accre.vanderbilt.
SLURM: Resource Management and Job Scheduling Software Advanced Computing Center for Research and Education www.accre.vanderbilt.edu Simple Linux Utility for Resource Management But it s also a job scheduler!
More informationHPC at IU Overview. Abhinav Thota Research Technologies Indiana University
HPC at IU Overview Abhinav Thota Research Technologies Indiana University What is HPC/cyberinfrastructure? Why should you care? Data sizes are growing Need to get to the solution faster Compute power is
More informationRa - Batch Scripts. Timothy H. Kaiser, Ph.D. tkaiser@mines.edu
Ra - Batch Scripts Timothy H. Kaiser, Ph.D. tkaiser@mines.edu Jobs on Ra are Run via a Batch System Ra is a shared resource Purpose: Give fair access to all users Have control over where jobs are run Set
More informationTechnical Guide to ULGrid
Technical Guide to ULGrid Ian C. Smith Computing Services Department September 4, 2007 1 Introduction This document follows on from the User s Guide to Running Jobs on ULGrid using Condor-G [1] and gives
More informationGC3: Grid Computing Competence Center Cluster computing, I Batch-queueing systems
GC3: Grid Computing Competence Center Cluster computing, I Batch-queueing systems Riccardo Murri, Sergio Maffioletti Grid Computing Competence Center, Organisch-Chemisches Institut, University of Zurich
More informationNew High-performance computing cluster: PAULI. Sascha Frick Institute for Physical Chemistry
New High-performance computing cluster: PAULI Sascha Frick Institute for Physical Chemistry 02/05/2012 Sascha Frick (PHC) HPC cluster pauli 02/05/2012 1 / 24 Outline 1 About this seminar 2 New Hardware
More informationIntroduction to HPC Workshop. Center for e-research (eresearch@nesi.org.nz)
Center for e-research (eresearch@nesi.org.nz) Outline 1 About Us About CER and NeSI The CS Team Our Facilities 2 Key Concepts What is a Cluster Parallel Programming Shared Memory Distributed Memory 3 Using
More informationJuropa. Batch Usage Introduction. May 2014 Chrysovalantis Paschoulas c.paschoulas@fz-juelich.de
Juropa Batch Usage Introduction May 2014 Chrysovalantis Paschoulas c.paschoulas@fz-juelich.de Batch System Usage Model A Batch System: monitors and controls the resources on the system manages and schedules
More informationRocoto. HWRF Python Scripts Training Miami, FL November 19, 2015
Rocoto HWRF Python Scripts Training Miami, FL November 19, 2015 Outline Introduction to Rocoto How it works Overview and description of XML Effectively using Rocoto (run, boot, stat, check, rewind, logs)
More informationSimplest Scalable Architecture
Simplest Scalable Architecture NOW Network Of Workstations Many types of Clusters (form HP s Dr. Bruce J. Walker) High Performance Clusters Beowulf; 1000 nodes; parallel programs; MPI Load-leveling Clusters
More informationCloud Computing. Up until now
Cloud Computing Lecture 3 Grid Schedulers: Condor, Sun Grid Engine 2010-2011 Introduction. Up until now Definition of Cloud Computing. Grid Computing: Schedulers: Condor architecture. 1 Summary Condor:
More informationParallel Computing using MATLAB Distributed Compute Server ZORRO HPC
Parallel Computing using MATLAB Distributed Compute Server ZORRO HPC Goals of the session Overview of parallel MATLAB Why parallel MATLAB? Multiprocessing in MATLAB Parallel MATLAB using the Parallel Computing
More informationIntroduction to Linux and Cluster Basics for the CCR General Computing Cluster
Introduction to Linux and Cluster Basics for the CCR General Computing Cluster Cynthia Cornelius Center for Computational Research University at Buffalo, SUNY 701 Ellicott St Buffalo, NY 14203 Phone: 716-881-8959
More informationJob scheduler details
Job scheduler details Advanced Computing Center for Research & Education (ACCRE) Job scheduler details 1 / 25 Outline 1 Batch queue system overview 2 Torque and Moab 3 Submitting jobs (ACCRE) Job scheduler
More informationRA MPI Compilers Debuggers Profiling. March 25, 2009
RA MPI Compilers Debuggers Profiling March 25, 2009 Examples and Slides To download examples on RA 1. mkdir class 2. cd class 3. wget http://geco.mines.edu/workshop/class2/examples/examples.tgz 4. tar
More informationLab 1 Beginning C Program
Lab 1 Beginning C Program Overview This lab covers the basics of compiling a basic C application program from a command line. Basic functions including printf() and scanf() are used. Simple command line
More informationBatch Systems. provide a mechanism for submitting, launching, and tracking jobs on a shared resource
PBS INTERNALS PBS & TORQUE PBS (Portable Batch System)-software system for managing system resources on workstations, SMP systems, MPPs and vector computers. It was based on Network Queuing System (NQS)
More informationSLURM Workload Manager
SLURM Workload Manager What is SLURM? SLURM (Simple Linux Utility for Resource Management) is the native scheduler software that runs on ASTI's HPC cluster. Free and open-source job scheduler for the Linux
More informationHigh Performance Computing with Sun Grid Engine on the HPSCC cluster. Fernando J. Pineda
High Performance Computing with Sun Grid Engine on the HPSCC cluster Fernando J. Pineda HPSCC High Performance Scientific Computing Center (HPSCC) " The Johns Hopkins Service Center in the Dept. of Biostatistics
More informationHTCondor within the European Grid & in the Cloud
HTCondor within the European Grid & in the Cloud Andrew Lahiff STFC Rutherford Appleton Laboratory HEPiX 2015 Spring Workshop, Oxford The Grid Introduction Computing element requirements Job submission
More informationExtreme computing lab exercises Session one
Extreme computing lab exercises Session one Miles Osborne (original: Sasa Petrovic) October 23, 2012 1 Getting started First you need to access the machine where you will be doing all the work. Do this
More informationRunning on Blue Gene/Q at Argonne Leadership Computing Facility (ALCF)
Running on Blue Gene/Q at Argonne Leadership Computing Facility (ALCF) ALCF Resources: Machines & Storage Mira (Production) IBM Blue Gene/Q 49,152 nodes / 786,432 cores 768 TB of memory Peak flop rate:
More informationTo connect to the cluster, simply use a SSH or SFTP client to connect to:
RIT Computer Engineering Cluster The RIT Computer Engineering cluster contains 12 computers for parallel programming using MPI. One computer, cluster-head.ce.rit.edu, serves as the master controller or
More informationThirty Useful Unix Commands
Leaflet U5 Thirty Useful Unix Commands Last revised April 1997 This leaflet contains basic information on thirty of the most frequently used Unix Commands. It is intended for Unix beginners who need a
More informationHPCC - Hrothgar Getting Started User Guide MPI Programming
HPCC - Hrothgar Getting Started User Guide MPI Programming High Performance Computing Center Texas Tech University HPCC - Hrothgar 2 Table of Contents 1. Introduction... 3 2. Setting up the environment...
More informationBeginners Shell Scripting for Batch Jobs
Beginners Shell Scripting for Batch Jobs Evan Bollig and Geoffrey Womeldorff Before we begin... Everyone please visit this page for example scripts and grab a crib sheet from the front http://www.scs.fsu.edu/~bollig/techseries
More informationHPCC USER S GUIDE. Version 1.2 July 2012. IITS (Research Support) Singapore Management University. IITS, Singapore Management University Page 1 of 35
HPCC USER S GUIDE Version 1.2 July 2012 IITS (Research Support) Singapore Management University IITS, Singapore Management University Page 1 of 35 Revision History Version 1.0 (27 June 2012): - Modified
More informationThe SUN ONE Grid Engine BATCH SYSTEM
The SUN ONE Grid Engine BATCH SYSTEM Juan Luis Chaves Sanabria Centro Nacional de Cálculo Científico (CeCalCULA) Latin American School in HPC on Linux Cluster October 27 November 07 2003 What is SGE? Is
More informationIntroduction to SDSC systems and data analytics software packages "
Introduction to SDSC systems and data analytics software packages " Mahidhar Tatineni (mahidhar@sdsc.edu) SDSC Summer Institute August 05, 2013 Getting Started" System Access Logging in Linux/Mac Use available
More informationHands-On UNIX Exercise:
Hands-On UNIX Exercise: This exercise takes you around some of the features of the shell. Even if you don't need to use them all straight away, it's very useful to be aware of them and to know how to deal
More informationParallel Processing using the LOTUS cluster
Parallel Processing using the LOTUS cluster Alison Pamment / Cristina del Cano Novales JASMIN/CEMS Workshop February 2015 Overview Parallelising data analysis LOTUS HPC Cluster Job submission on LOTUS
More informationUsing the Yale HPC Clusters
Using the Yale HPC Clusters Stephen Weston Robert Bjornson Yale Center for Research Computing Yale University Dec 2015 To get help Send an email to: hpc@yale.edu Read documentation at: http://research.computing.yale.edu/hpc-support
More informationGRID workload management system and CMS fall production. Massimo Sgaravatto INFN Padova
GRID workload management system and CMS fall production Massimo Sgaravatto INFN Padova What do we want to implement (simplified design) Master chooses in which resources the jobs must be submitted Condor-G
More informationMSU Tier 3 Usage and Troubleshooting. James Koll
MSU Tier 3 Usage and Troubleshooting James Koll Overview Dedicated computing for MSU ATLAS members Flexible user environment ~500 job slots of various configurations ~150 TB disk space 2 Condor commands
More informationJob Scheduling Explained More than you ever want to know about how jobs get scheduled on WestGrid systems...
Job Scheduling Explained More than you ever want to know about how jobs get scheduled on WestGrid systems... Martin Siegert, SFU Cluster Myths There are so many jobs in the queue - it will take ages until
More informationUnix Scripts and Job Scheduling
Unix Scripts and Job Scheduling Michael B. Spring Department of Information Science and Telecommunications University of Pittsburgh spring@imap.pitt.edu http://www.sis.pitt.edu/~spring Overview Shell Scripts
More informationUsing SVN to Manage Source RTL
Using SVN to Manage Source RTL CS250 Tutorial 1 (Version 092509a) September 25, 2009 Yunsup Lee In this tutorial you will gain experience using the Subversion (SVN) to manage your source RTL and code.
More informationHow To Run A Steady Case On A Creeper
Crash Course Introduction to OpenFOAM Artur Lidtke University of Southampton akl1g09@soton.ac.uk November 4, 2014 Artur Lidtke Crash Course Introduction to OpenFOAM 1 / 32 What is OpenFOAM? Using OpenFOAM
More informationThe CNMS Computer Cluster
The CNMS Computer Cluster This page describes the CNMS Computational Cluster, how to access it, and how to use it. Introduction (2014) The latest block of the CNMS Cluster (2010) Previous blocks of the
More informationMatlab on a Supercomputer
Matlab on a Supercomputer Shelley L. Knuth Research Computing April 9, 2015 Outline Description of Matlab and supercomputing Interactive Matlab jobs Non-interactive Matlab jobs Parallel Computing Slides
More informationThe RWTH Compute Cluster Environment
The RWTH Compute Cluster Environment Tim Cramer 11.03.2013 Source: D. Both, Bull GmbH Rechen- und Kommunikationszentrum (RZ) How to login Frontends cluster.rz.rwth-aachen.de cluster-x.rz.rwth-aachen.de
More informationORACLE NOSQL DATABASE HANDS-ON WORKSHOP Cluster Deployment and Management
ORACLE NOSQL DATABASE HANDS-ON WORKSHOP Cluster Deployment and Management Lab Exercise 1 Deploy 3x3 NoSQL Cluster into single Datacenters Objective: Learn from your experience how simple and intuitive
More informationMFCF Grad Session 2015
MFCF Grad Session 2015 Agenda Introduction Help Centre and requests Dept. Grad reps Linux clusters using R with MPI Remote applications Future computing direction Technical question and answer period MFCF
More informationAdvanced Techniques with Newton. Gerald Ragghianti Advanced Newton workshop Sept. 22, 2011
Advanced Techniques with Newton Gerald Ragghianti Advanced Newton workshop Sept. 22, 2011 Workshop Goals Gain independence Executing your work Finding Information Fixing Problems Optimizing Effectiveness
More informationBiowulf2 Training Session
Biowulf2 Training Session 9 July 2015 Slides at: h,p://hpc.nih.gov/docs/b2training.pdf HPC@NIH website: h,p://hpc.nih.gov System hardware overview What s new/different The batch system & subminng jobs
More informationUsing NeSI HPC Resources. NeSI Computational Science Team (support@nesi.org.nz)
NeSI Computational Science Team (support@nesi.org.nz) Outline 1 About Us About NeSI Our Facilities 2 Using the Cluster Suitable Work What to expect Parallel speedup Data Getting to the Login Node 3 Submitting
More informationParallel Programming for Multi-Core, Distributed Systems, and GPUs Exercises
Parallel Programming for Multi-Core, Distributed Systems, and GPUs Exercises Pierre-Yves Taunay Research Computing and Cyberinfrastructure 224A Computer Building The Pennsylvania State University University
More informationThe Asterope compute cluster
The Asterope compute cluster ÅA has a small cluster named asterope.abo.fi with 8 compute nodes Each node has 2 Intel Xeon X5650 processors (6-core) with a total of 24 GB RAM 2 NVIDIA Tesla M2050 GPGPU
More informationHP Operations Manager Software for Windows Integration Guide
HP Operations Manager Software for Windows Integration Guide This guide documents the facilities to integrate EnterpriseSCHEDULE into HP Operations Manager Software for Windows (formerly known as HP OpenView
More informationOn-demand (Pay-per-Use) HPC Service Portal
On-demand (Pay-per-Use) Portal Wang Junhong INTRODUCTION High Performance Computing, Computer Centre The Service Portal is a key component of the On-demand (pay-per-use) HPC service delivery. The Portal,
More informationHow To Run A Tompouce Cluster On An Ipra (Inria) 2.5.5 (Sun) 2 (Sun Geserade) 2-5.4 (Sun-Ge) 2/5.2 (
Running Hadoop and Stratosphere jobs on TomPouce cluster 16 October 2013 TomPouce cluster TomPouce is a cluster of 20 calcula@on nodes = 240 cores Located in the Inria Turing building (École Polytechnique)
More informationResource Management and Job Scheduling
Resource Management and Job Scheduling Jenett Tillotson Senior Cluster System Administrator Indiana University May 18 18-22 May 2015 1 Resource Managers Keep track of resources Nodes: CPUs, disk, memory,
More informationPBSPro scheduling. PBS overview Qsub command: resource requests. Queues a7ribu8on. Fairshare. Backfill Jobs submission.
PBSPro scheduling PBS overview Qsub command: resource requests Queues a7ribu8on Fairshare Backfill Jobs submission 9 mai 03 PBS PBS overview 9 mai 03 PBS PBS organiza5on: daemons frontend compute nodes
More informationHigh Performance Computing Facility Specifications, Policies and Usage. Supercomputer Project. Bibliotheca Alexandrina
High Performance Computing Facility Specifications, Policies and Usage Supercomputer Project Bibliotheca Alexandrina Bibliotheca Alexandrina 1/16 Topics Specifications Overview Site Policies Intel Compilers
More informationNotes on the SNOW/Rmpi R packages with OpenMPI and Sun Grid Engine
Notes on the SNOW/Rmpi R packages with OpenMPI and Sun Grid Engine Last updated: 6/2/2008 4:43PM EDT We informally discuss the basic set up of the R Rmpi and SNOW packages with OpenMPI and the Sun Grid
More informationHow to Run Parallel Jobs Efficiently
How to Run Parallel Jobs Efficiently Shao-Ching Huang High Performance Computing Group UCLA Institute for Digital Research and Education May 9, 2013 1 The big picture: running parallel jobs on Hoffman2
More informationThe XSEDE Global Federated File System (GFFS) - Breaking Down Barriers to Secure Resource Sharing
December 19, 2013 The XSEDE Global Federated File System (GFFS) - Breaking Down Barriers to Secure Resource Sharing Andrew Grimshaw, University of Virginia Co-architect XSEDE The complexity of software
More informationThe Maui High Performance Computing Center Department of Defense Supercomputing Resource Center (MHPCC DSRC) Hadoop Implementation on Riptide - -
The Maui High Performance Computing Center Department of Defense Supercomputing Resource Center (MHPCC DSRC) Hadoop Implementation on Riptide - - Hadoop Implementation on Riptide 2 Table of Contents Executive
More informationStanford HPC Conference. Panasas Storage System Integration into a Cluster
Stanford HPC Conference Panasas Storage System Integration into a Cluster David Yu Industry Verticals Panasas Inc. Steve Jones Technology Operations Manager Institute for Computational and Mathematical
More informationLinux Firewalls (Ubuntu IPTables) II
Linux Firewalls (Ubuntu IPTables) II Here we will complete the previous firewall lab by making a bridge on the Ubuntu machine, to make the Ubuntu machine completely control the Internet connection on the
More informationSubmitting batch jobs Slurm on ecgate. Xavi Abellan xavier.abellan@ecmwf.int User Support Section
Submitting batch jobs Slurm on ecgate Xavi Abellan xavier.abellan@ecmwf.int User Support Section Slide 1 Outline Interactive mode versus Batch mode Overview of the Slurm batch system on ecgate Batch basic
More informationUsing the Millipede cluster - I
Using the Millipede cluster - I Fokke Dijkstra, Bob Dröge High Performance Computing and Visualisation group Donald Smits Centre for Information Technology General introduction Course aimed at beginners
More informationA High Performance Computing Scheduling and Resource Management Primer
LLNL-TR-652476 A High Performance Computing Scheduling and Resource Management Primer D. H. Ahn, J. E. Garlick, M. A. Grondona, D. A. Lipari, R. R. Springmeyer March 31, 2014 Disclaimer This document was
More informationSummary. Load and Open GaussView to start example
Summary This document describes in great detail how to navigate the Linux Red Hat Terminal to bring up GaussView, use GaussView to create a simple atomic or molecular simulation input file, and then use
More information