High-Performance Computing

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "High-Performance Computing"

Transcription

1 High-Performance Computing Windows, Matlab and the HPC Dr. Leigh Brookshaw Dept. of Maths and Computing, USQ 1 The HPC Architecture 30 Sun boxes or nodes Each node has 2 x 2.4GHz AMD CPUs with 4 Cores each and 16GB RAM. Theoretically possible to run 240 independent simultaneous jobs. One extra node is the controller or administration node. Access to the HPC is via the administration node. One extra node is the Input/Output node it is the disk controller for the HPC. All disk space (5 Tb) is controlled by this node. One extra node for Matlab clients The disk space is visible to all nodes. 2.9Tb backed up, 2.1Tb temporary space not backed up and cleared at reboot. One extra machine, for code testing and interactive jobs, has 4 x 2.4GHz AMD CPUs with 6 Cores each and 64GB. 2

2 The HPC Architecture 3 The HPC Architecture CPU speed is equivalent to a workstation Computational power is derived from using more than one node/core. There is no advantage if you run one job on one core. 4

3 Connecting to the HPC habeus only for code testing, some interactive jobs and some batch jobs. usqhpcio normally don t need to connect to it. Controls the disks should never run jobs on this node. usqhpc main access point to the HPC. Contains the queues for submitting jobs. usqhpcm for Matlab jobs using the Parallel Toolbox. The only access to the HPC is via a Secure Shell from the USQ network. This ensures all communication to the HPC is encrypted and secure. The HPC uses RedHat Linux. Most interaction with the HPC does not require a detailed knowledge of the Unix/Linux command line about 10 conmmands 5 Connecting to the HPC Windows Utilities PuTTY creates an SSH connection and provides a command line interface to a remote machine. WinSCP provides a traditional Windows interface for copying files between the local machine and the remote machine. Uses SSH to ensure all communication is encrypted notepad++ a good all purpose text/code editor that recognises Windows, Unix and MacOS text files. 6

4 PuTTY Main Window Need the name of the machine you wish to connect to 7 PuTTY First Connect First time PuTTY connects to a machine it will ask you whether it should download the remote host s identifying key answer Yes 8

5 PuTTY Command Line Window Need to enter your HPC username and password. 9 WinSCP Connecting to remote host Need to enter the machine name and your username. 10

6 WinSCP Preferences In the preferences you can specify which editor to use. Remote files are downloaded edited locally then uploaded. Downloading and uploading of files is done automatically. 11 WinSCP Main Window 12

7 Types of Jobs Two basic types of jobs are run on multiple computing nodes Distributed Jobs: Also called Course Grained jobs Each process is completely independent of each other Little or no communication between processes Examples: Parameter space search, running the same program repeatedly with different input parameters, &c. Parallel Jobs: Also called Fine Grained jobs Each process deals with a part of the problem Communication synchronisation required between processes. Example: Processing of large data sets that will not fit on one node, computational domains that need to be split across nodes, CFD &c. 13 Distributed Job Example Paradigm Master-Slave processing 14

8 Parallel Job Example Paradigm Peer-Peer processing 15 HPC Job Submission Jobs are run via a Batch System PBS (Portable Batch System) A batch system requires jobs to run unsupervised! Jobs submitted on batch queue batch system starts job running when requested resources are available Requested Resources: Number of nodes Number of cores per node Memory per process Total amount of memory for the job Maximum amount of time... Jobs are submitted via a Shell Script or via a Matlab script Shell Script examples are available on the HPC web site 16

9 Shell Script Example #Select resources ##### #PBS -N Test-Matlab #PBS -l nodes=7:ppn=3 ##### Queue ##### #PBS -q standard ##### Mail Options ##### #PBS -m bea #PBS -M ##### Change to current working directory ##### cd /home/mcsci/leighb/test ##### Execute Program ##### /usr/local/bin/matlab -nodisplay -nodesktop -nosplash < driver.m 17 Using Matlab 18

10 Matlab s Parallel Toolbox Provides the infrastructure scripts/commands for Parallel or Distributed computing Ability to create Matlab Workers (a running instance of Matlab). Ability to assign tasks to Workers (pass a script to run on a worker). Ability to communicate between Workers (running scripts can communicate with each other). Provides a Local Scheduler that will allow you to start one worker per core on one node. Maximum number of workers is 8 19 Matlab s Distributed Computing Server Provides the Schedular to create Workers on other nodes. Accessed via the Parallel Toolbox when requesting Workers Currently the HPC has a license for a total of 64 simultaneous Workers. 20

11 Matlab Client Script Uses one Matlab license! Uses one Distributed Computing Toolbox license! Requests resources from the Scheduler the number of Workers required &c. Client script distributes tasks to the Workers. Client script can run on the HPC (usqhpcm) or on your own machine Client machine must be able to connect to the HPC. 21 Matlab Distributed Processing Minimalist Example running on USQHPCM sched = findresource( scheduler, type, torque ); set(sched, HasSharedFilesystem, true); set(sched, DataLocation, /sandisk1/leighb ) set(sched, RshCommand, ssh ); job=createjob(sched); set(job, PathDependencies, { /home/mcsci/leighb/test }); for i=1:max 3, {nsim} ); end submit(job); waitforstate(job); results = getalloutputarguments(job); 22

12 Matlab Distributed Processing torque is the name of the Scheduler to use with the HPC s PBS batch system. createjob() create a new distributed job to submit to the scheduler createtask(): specify the tasks for the job. One task to one worker. A task is a Matlab function to run. submit(): queue the job on the PBS batch system using the torque scheduler. Each worker appears as a separate queued job on the PBS default queue. waitforstate(): wait for the job to complete. Can timeout or wait for specific tasks to finish. getalloutputarguments(): get the return values from the job. 23 Matlab Distributed Processing Required Settings HasSharedFilesystem All the nodes can see the user s home folder on the HPC. Set to true DataLocation Matlab s book-keeping location. Place where Task output/input can be stored by Workers. RshCommand The command Matlab must use to communicate between the Client and the Workers. PathDependencies The paths to all the scripts used in this job so all the Workers can find them. 24

13 Matlab Distributed Processing Comments Each Worker appears as a submitted job on the PBS queue Workers will be distributed by the PBS system set(sched, SubmitArguments, -q long ); Additional arguments to use when submitting Tasks to the PBS queue. Most common use is to change the queues. 25 Distributed versus Parallel Distributed Job Matlab sessions called Workers Workers cannot communicate with each other. Define any number of tasks (different or the same) in a job Each Task is queued on the PBS system Tasks need not run simultaneously assigned to Workers as they become available. Workers can run several tasks in a job Parallel Job Matlab sessions called Labs Labs can communicate with each other. Define one task for the job duplicates are run on all Labs requested The Job is queued on the PBS system Tasks run simultaneously on as many Labs available at runtime. The start of the job may have to wait until the requested number of Labs is available. 26

14 Matlab parallel Processing Explicit Example sched = findresource( scheduler, type, torque ); set(sched, HasSharedFilesystem, true); set(sched, DataLocation, /sandisk1/leighb ) set(sched, RshCommand, ssh ); pjob=createparalleljob(sched); set(pjob, PathDependencies, { /home/mcsci/leighb/test }); set(pjob, MaximumNumberOfWorkers, 30) set(pjob, MinimumNumberOfWorkers, 20) t = 3, {nsim} ); submit(pjob); waitforstate(pjob); results = getalloutputarguments(pjob); 27 Matlab Parallel Processing Comments on Explicit example One task only it is repeated on all Labs! Only one job is submitted on the PBS queue. Matlab defaults to requesting from PBS one node for each lab! If you need more than 30 Labs you must explicitly specify resources required set(sched, ResourceTemplate, -l nodes=30:ppn=2 ); \\ set(pjob, MaximumNumberOfWorkers, 60) set(pjob, MinimumNumberOfWorkers, 60) Currently only have a license for 64 workers at any one time! 28

15 Explicit Lab Communication and Synchronisation numlabs returns the number of Labs in the current job. labindex returns the index of Lab. Value will be different for each lab. labsend send data to the specified Lab. labreceive block and read data from a specific Lab. labprobe check if data is available from a specific Lab. labbarrier block execution until all labs reach this call Matlab Parallel Processing Letting Matlab do the work! sched = findresource( scheduler, type, torque ); set(sched, HasSharedFilesystem, true); set(sched, DataLocation, /sandisk1/leighb ) set(sched, RshCommand, ssh ); set(sched, RcpCommand, scp ); matlabpool(sched,24); parfor i=1:64 result(i,:) = distance(100000); end matlabpool close; 30

16 Matlab Parallel Processing Points to note matlabpool start 24 Labs for this job. One job appears in the PBS queue with requested resources 24 nodes. parfor distribute the contents of the loop to the Labs in the pool. Each Lab works on one iteration of the loop! The first 24 iterations calculated simultaneously one on each Lab. Loop iterations are not done in loop order but in Parallel results will appear out of order unless stored in an explicitly indexed array! One iteration of a loop cannot depend on a previous iteration. Pool of Labs remain running and available for tasks until the Pool is explicitly closed. 31 Matlab Parallel Processing Single Program Multiple Data (spmd) Interleaving of serial and parallel computing in the one client script Use Matlab smpd... end blocks Parallel computing within the smpd block serial outside! Identical code runs on each Lab different data Useful for running the same program on different data sets when communication and synchronisation is required! The Lab data sets may be part of a large distributed data set! 32

17 Matlab Parallel Processing SMPD Example matlabpool(sched,24); spmd R1 = rand(240); Z1 = zeroes(240); Z2 = codistributed(z1); Z3 = getlocalpart(z2); Z4 = codistributed.rand(100000,24) Z5 = gather(z2,1); end matlabpool close; 33 Matlab Parallel Processing SMPD Example... R1 is a different array replicated on each Lab. Z1 is an array replicated on each Lab. Z2 is a codistributed array one segment of Z1 to each Lab. Default segmentation is by the last non-unary dimension columns in this case. Z3 contains the local Lab. part of Z2. Create a codistributed array use if the distributed array is too large to replicate on each Lab. Z5 in Lab 1 contains the reconstructed Z2. Without the 1 all Labs contain the reconstructed array. 34

18 Matlab Parallel Processing SMPD... Many Matlab functions are capable of working with codistributed arrays Elementary Array operations +, -, *, /, /, dot variants, &c. Elementary Matrix operations find, diag, reshape, size, sort, is*, &c. Matrix functions Eigen, inverse, LU factorization, SVD, Norms, &c. Elementary trig., log, hyperbolic functions, &c. help codistributed/functionname For-loops on codistributed arrays can only loop over the parts local to each Lab! 35

Parallel Computing using MATLAB Distributed Compute Server ZORRO HPC

Parallel Computing using MATLAB Distributed Compute Server ZORRO HPC Parallel Computing using MATLAB Distributed Compute Server ZORRO HPC Goals of the session Overview of parallel MATLAB Why parallel MATLAB? Multiprocessing in MATLAB Parallel MATLAB using the Parallel Computing

More information

Introductory Tutorial to Parallel and Distributed Computing Tools of cluster.tigem.it

Introductory Tutorial to Parallel and Distributed Computing Tools of cluster.tigem.it Introductory Tutorial to Parallel and Distributed Computing Tools of cluster.tigem.it A Computer Cluster is a group of networked computers, working together closely The computer are called nodes Cluster

More information

Linux für bwgrid. Sabine Richling, Heinz Kredel. Universitätsrechenzentrum Heidelberg Rechenzentrum Universität Mannheim. 27.

Linux für bwgrid. Sabine Richling, Heinz Kredel. Universitätsrechenzentrum Heidelberg Rechenzentrum Universität Mannheim. 27. Linux für bwgrid Sabine Richling, Heinz Kredel Universitätsrechenzentrum Heidelberg Rechenzentrum Universität Mannheim 27. June 2011 Richling/Kredel (URZ/RUM) Linux für bwgrid FS 2011 1 / 33 Introduction

More information

Introduction to Matlab Distributed Computing Server (MDCS) Dan Mazur and Pier-Luc St-Onge guillimin@calculquebec.ca December 1st, 2015

Introduction to Matlab Distributed Computing Server (MDCS) Dan Mazur and Pier-Luc St-Onge guillimin@calculquebec.ca December 1st, 2015 Introduction to Matlab Distributed Computing Server (MDCS) Dan Mazur and Pier-Luc St-Onge guillimin@calculquebec.ca December 1st, 2015 1 Partners and sponsors 2 Exercise 0: Login and Setup Ubuntu login:

More information

Matlab on a Supercomputer

Matlab on a Supercomputer Matlab on a Supercomputer Shelley L. Knuth Research Computing April 9, 2015 Outline Description of Matlab and supercomputing Interactive Matlab jobs Non-interactive Matlab jobs Parallel Computing Slides

More information

Introduction to MSI* for PubH 8403

Introduction to MSI* for PubH 8403 Introduction to MSI* for PubH 8403 Sep 30, 2015 Nancy Rowe *The Minnesota Supercomputing Institute for Advanced Computational Research Overview MSI at a Glance MSI Resources Access System Access - Physical

More information

MFCF Grad Session 2015

MFCF Grad Session 2015 MFCF Grad Session 2015 Agenda Introduction Help Centre and requests Dept. Grad reps Linux clusters using R with MPI Remote applications Future computing direction Technical question and answer period MFCF

More information

Chapter 2: Getting Started

Chapter 2: Getting Started Chapter 2: Getting Started Once Partek Flow is installed, Chapter 2 will take the user to the next stage and describes the user interface and, of note, defines a number of terms required to understand

More information

An Introduction to High Performance Computing in the Department

An Introduction to High Performance Computing in the Department An Introduction to High Performance Computing in the Department Ashley Ford & Chris Jewell Department of Statistics University of Warwick October 30, 2012 1 Some Background 2 How is Buster used? 3 Software

More information

Using the Yale HPC Clusters

Using the Yale HPC Clusters Using the Yale HPC Clusters Stephen Weston Robert Bjornson Yale Center for Research Computing Yale University Oct 2015 To get help Send an email to: hpc@yale.edu Read documentation at: http://research.computing.yale.edu/hpc-support

More information

Cluster@WU User s Manual

Cluster@WU User s Manual Cluster@WU User s Manual Stefan Theußl Martin Pacala September 29, 2014 1 Introduction and scope At the WU Wirtschaftsuniversität Wien the Research Institute for Computational Methods (Forschungsinstitut

More information

Miami University RedHawk Cluster Working with batch jobs on the Cluster

Miami University RedHawk Cluster Working with batch jobs on the Cluster Miami University RedHawk Cluster Working with batch jobs on the Cluster The RedHawk cluster is a general purpose research computing resource available to support the research community at Miami University.

More information

Parallel Computing with MATLAB

Parallel Computing with MATLAB Parallel Computing with MATLAB Scott Benway Senior Account Manager Jiro Doke, Ph.D. Senior Application Engineer 2013 The MathWorks, Inc. 1 Acceleration Strategies Applied in MATLAB Approach Options Best

More information

MATLAB Distributed Computing Server Cloud Center User s Guide

MATLAB Distributed Computing Server Cloud Center User s Guide MATLAB Distributed Computing Server Cloud Center User s Guide How to Contact MathWorks Latest news: Sales and services: User community: Technical support: www.mathworks.com www.mathworks.com/sales_and_services

More information

SSH Connections MACs the MAC XTerm application can be used to create an ssh connection, no utility is needed.

SSH Connections MACs the MAC XTerm application can be used to create an ssh connection, no utility is needed. Overview of MSU Compute Servers The DECS Linux based compute servers are well suited for programs that are too slow to run on typical desktop computers but do not require the power of supercomputers. The

More information

Using WestGrid. Patrick Mann, Manager, Technical Operations Jan.15, 2014

Using WestGrid. Patrick Mann, Manager, Technical Operations Jan.15, 2014 Using WestGrid Patrick Mann, Manager, Technical Operations Jan.15, 2014 Winter 2014 Seminar Series Date Speaker Topic 5 February Gino DiLabio Molecular Modelling Using HPC and Gaussian 26 February Jonathan

More information

Bringing Big Data Modelling into the Hands of Domain Experts

Bringing Big Data Modelling into the Hands of Domain Experts Bringing Big Data Modelling into the Hands of Domain Experts David Willingham Senior Application Engineer MathWorks david.willingham@mathworks.com.au 2015 The MathWorks, Inc. 1 Data is the sword of the

More information

Beyond Windows: Using the Linux Servers and the Grid

Beyond Windows: Using the Linux Servers and the Grid Beyond Windows: Using the Linux Servers and the Grid Topics Linux Overview How to Login & Remote Access Passwords Staying Up-To-Date Network Drives Server List The Grid Useful Commands Linux Overview Linux

More information

MATLAB Distributed Computing Server Installation Guide. R2012a

MATLAB Distributed Computing Server Installation Guide. R2012a MATLAB Distributed Computing Server Installation Guide R2012a How to Contact MathWorks www.mathworks.com Web comp.soft-sys.matlab Newsgroup www.mathworks.com/contact_ts.html Technical Support suggest@mathworks.com

More information

Parallel Computing with Mathematica UVACSE Short Course

Parallel Computing with Mathematica UVACSE Short Course UVACSE Short Course E Hall 1 1 University of Virginia Alliance for Computational Science and Engineering uvacse@virginia.edu October 8, 2014 (UVACSE) October 8, 2014 1 / 46 Outline 1 NX Client for Remote

More information

Secure Shell. The Protocol

Secure Shell. The Protocol Usually referred to as ssh The name is used for both the program and the protocol ssh is an extremely versatile network program data encryption and compression terminal access to remote host file transfer

More information

Installing a Symantec Backup Exec Agent on a SnapScale Cluster X2 Node or SnapServer DX1 or DX2. Summary

Installing a Symantec Backup Exec Agent on a SnapScale Cluster X2 Node or SnapServer DX1 or DX2. Summary Technical Bulletin Application Note April 2013 Installing a Symantec Backup Exec Agent on a SnapScale Cluster X2 Node or SnapServer DX1 or DX2 Summary This application note describes how to install the

More information

1. Product Information

1. Product Information ORIXCLOUD BACKUP CLIENT USER MANUAL LINUX 1. Product Information Product: Orixcloud Backup Client for Linux Version: 4.1.7 1.1 System Requirements Linux (RedHat, SuSE, Debian and Debian based systems such

More information

Online Backup Client User Manual Linux

Online Backup Client User Manual Linux Online Backup Client User Manual Linux 1. Product Information Product: Online Backup Client for Linux Version: 4.1.7 1.1 System Requirements Operating System Linux (RedHat, SuSE, Debian and Debian based

More information

LSKA 2010 Survey Report Job Scheduler

LSKA 2010 Survey Report Job Scheduler LSKA 2010 Survey Report Job Scheduler Graduate Institute of Communication Engineering {r98942067, r98942112}@ntu.edu.tw March 31, 2010 1. Motivation Recently, the computing becomes much more complex. However,

More information

HPCC - Hrothgar Getting Started User Guide

HPCC - Hrothgar Getting Started User Guide HPCC - Hrothgar Getting Started User Guide Transfer files High Performance Computing Center Texas Tech University HPCC - Hrothgar 2 Table of Contents Transferring files... 3 1.1 Transferring files using

More information

RecoveryVault Express Client User Manual

RecoveryVault Express Client User Manual For Linux distributions Software version 4.1.7 Version 2.0 Disclaimer This document is compiled with the greatest possible care. However, errors might have been introduced caused by human mistakes or by

More information

Online Backup Linux Client User Manual

Online Backup Linux Client User Manual Online Backup Linux Client User Manual Software version 4.0.x For Linux distributions August 2011 Version 1.0 Disclaimer This document is compiled with the greatest possible care. However, errors might

More information

Operating Systems OBJECTIVES 7.1 DEFINITION. Chapter 7. Note:

Operating Systems OBJECTIVES 7.1 DEFINITION. Chapter 7. Note: Chapter 7 OBJECTIVES Operating Systems Define the purpose and functions of an operating system. Understand the components of an operating system. Understand the concept of virtual memory. Understand the

More information

Online Backup Client User Manual

Online Backup Client User Manual For Linux distributions Software version 4.1.7 Version 2.0 Disclaimer This document is compiled with the greatest possible care. However, errors might have been introduced caused by human mistakes or by

More information

High-Performance Reservoir Risk Assessment (Jacta Cluster)

High-Performance Reservoir Risk Assessment (Jacta Cluster) High-Performance Reservoir Risk Assessment (Jacta Cluster) SKUA-GOCAD 2013.1 Paradigm 2011.3 With Epos 4.1 Data Management Configuration Guide 2008 2013 Paradigm Ltd. or its affiliates and subsidiaries.

More information

Introduction to Linux and Cluster Basics for the CCR General Computing Cluster

Introduction to Linux and Cluster Basics for the CCR General Computing Cluster Introduction to Linux and Cluster Basics for the CCR General Computing Cluster Cynthia Cornelius Center for Computational Research University at Buffalo, SUNY 701 Ellicott St Buffalo, NY 14203 Phone: 716-881-8959

More information

Tackling Big Data with MATLAB Adam Filion Application Engineer MathWorks, Inc.

Tackling Big Data with MATLAB Adam Filion Application Engineer MathWorks, Inc. Tackling Big Data with MATLAB Adam Filion Application Engineer MathWorks, Inc. 2015 The MathWorks, Inc. 1 Challenges of Big Data Any collection of data sets so large and complex that it becomes difficult

More information

Calcul Parallèle sous MATLAB

Calcul Parallèle sous MATLAB Calcul Parallèle sous MATLAB Journée Calcul Parallèle GPU/CPU du PEPI MACS Olivier de Mouzon INRA Gremaq Toulouse School of Economics Lundi 28 novembre 2011 Paris Présentation en grande partie fondée sur

More information

CASHNet Secure File Transfer Instructions

CASHNet Secure File Transfer Instructions CASHNet Secure File Transfer Instructions Copyright 2009, 2010 Higher One Payments, Inc. CASHNet, CASHNet Business Office, CASHNet Commerce Center, CASHNet SMARTPAY and all related logos and designs are

More information

The Asterope compute cluster

The Asterope compute cluster The Asterope compute cluster ÅA has a small cluster named asterope.abo.fi with 8 compute nodes Each node has 2 Intel Xeon X5650 processors (6-core) with a total of 24 GB RAM 2 NVIDIA Tesla M2050 GPGPU

More information

Uptime Infrastructure Monitor. Installation Guide

Uptime Infrastructure Monitor. Installation Guide Uptime Infrastructure Monitor Installation Guide This guide will walk through each step of installation for Uptime Infrastructure Monitor software on a Windows server. Uptime Infrastructure Monitor is

More information

Grid Engine Basics. Table of Contents. Grid Engine Basics Version 1. (Formerly: Sun Grid Engine)

Grid Engine Basics. Table of Contents. Grid Engine Basics Version 1. (Formerly: Sun Grid Engine) Grid Engine Basics (Formerly: Sun Grid Engine) Table of Contents Table of Contents Document Text Style Associations Prerequisites Terminology What is the Grid Engine (SGE)? Loading the SGE Module on Turing

More information

RenderStorm Cloud Render (Powered by Squidnet Software): Getting started.

RenderStorm Cloud Render (Powered by Squidnet Software): Getting started. Version 1.0 RenderStorm Cloud Render (Powered by Squidnet Software): Getting started. RenderStorm Cloud Render is an easy to use standalone application providing remote access, job submission, rendering,

More information

Introduction to Running Hadoop on the High Performance Clusters at the Center for Computational Research

Introduction to Running Hadoop on the High Performance Clusters at the Center for Computational Research Introduction to Running Hadoop on the High Performance Clusters at the Center for Computational Research Cynthia Cornelius Center for Computational Research University at Buffalo, SUNY 701 Ellicott St

More information

WinSCP PuTTY as an alternative to F-Secure July 11, 2006

WinSCP PuTTY as an alternative to F-Secure July 11, 2006 WinSCP PuTTY as an alternative to F-Secure July 11, 2006 Brief Summary of this Document F-Secure SSH Client 5.4 Build 34 is currently the Berkeley Lab s standard SSH client. It consists of three integrated

More information

Rsync-enabled NAS Hardware Compatibility List

Rsync-enabled NAS Hardware Compatibility List WHITEPAPER BackupAssist Version 5.1 www.backupassist.com Cortex I.T. Labs 2001-2008 2 Contents Introduction... 3 Hardware Setup Instructions... 3 QNAP TS-409... 3 Netgear ReadyNas NV+... 5 Drobo rev1...

More information

Deployment of Keepit for Windows

Deployment of Keepit for Windows Deployment of Keepit for Windows Keepit A/S October 13, 2010 1 Introduction When deploying Keepit in larger setups with many desktops and servers, installing Keepit individually on each computer is cumbersome

More information

Streamline Computing Linux Cluster User Training. ( Nottingham University)

Streamline Computing Linux Cluster User Training. ( Nottingham University) 1 Streamline Computing Linux Cluster User Training ( Nottingham University) 3 User Training Agenda System Overview System Access Description of Cluster Environment Code Development Job Schedulers Running

More information

Parallel Processing using the LOTUS cluster

Parallel Processing using the LOTUS cluster Parallel Processing using the LOTUS cluster Alison Pamment / Cristina del Cano Novales JASMIN/CEMS Workshop February 2015 Overview Parallelising data analysis LOTUS HPC Cluster Job submission on LOTUS

More information

MATLAB Distributed Computing Server System Administrator's Guide

MATLAB Distributed Computing Server System Administrator's Guide MATLAB Distributed Computing Server System Administrator's Guide R2015b How to Contact MathWorks Latest news: www.mathworks.com Sales and services: www.mathworks.com/sales_and_services User community:

More information

CycleServer Grid Engine Support Install Guide. version 1.25

CycleServer Grid Engine Support Install Guide. version 1.25 CycleServer Grid Engine Support Install Guide version 1.25 Contents CycleServer Grid Engine Guide 1 Administration 1 Requirements 1 Installation 1 Monitoring Additional OGS/SGE/etc Clusters 3 Monitoring

More information

Work Environment. David Tur HPC Expert. HPC Users Training September, 18th 2015

Work Environment. David Tur HPC Expert. HPC Users Training September, 18th 2015 Work Environment David Tur HPC Expert HPC Users Training September, 18th 2015 1. Atlas Cluster: Accessing and using resources 2. Software Overview 3. Job Scheduler 1. Accessing Resources DIPC technicians

More information

SSH and Basic Commands

SSH and Basic Commands SSH and Basic Commands In this tutorial we'll introduce you to SSH - a tool that allows you to send remote commands to your Web server - and show you some simple UNIX commands to help you manage your website.

More information

Quick Tutorial for Portable Batch System (PBS)

Quick Tutorial for Portable Batch System (PBS) Quick Tutorial for Portable Batch System (PBS) The Portable Batch System (PBS) system is designed to manage the distribution of batch jobs and interactive sessions across the available nodes in the cluster.

More information

File Transfers. Contents

File Transfers. Contents A File Transfers Contents Overview..................................................... A-2................................... A-2 General Switch Software Download Rules..................... A-3 Using

More information

PuTTY/Cygwin Tutorial. By Ben Meister Written for CS 23, Winter 2007

PuTTY/Cygwin Tutorial. By Ben Meister Written for CS 23, Winter 2007 PuTTY/Cygwin Tutorial By Ben Meister Written for CS 23, Winter 2007 This tutorial will show you how to set up and use PuTTY to connect to CS Department computers using SSH, and how to install and use the

More information

Visualization Cluster Getting Started

Visualization Cluster Getting Started Visualization Cluster Getting Started Contents 1 Introduction to the Visualization Cluster... 1 2 Visualization Cluster hardware and software... 2 3 Remote visualization session through VNC... 2 4 Starting

More information

Navigating the Rescue Mode for Linux

Navigating the Rescue Mode for Linux Navigating the Rescue Mode for Linux SUPPORT GUIDE DEDICATED SERVERS ABOUT THIS GUIDE This document will take you through the process of booting your Linux server into rescue mode to identify and fix the

More information

FileCruiser Backup & Restoring Guide

FileCruiser Backup & Restoring Guide FileCruiser Backup & Restoring Guide Version: 0.3 FileCruiser Model: VA2600/VR2600 with SR1 Date: JAN 27, 2015 1 Index Index... 2 Introduction... 3 Backup Requirements... 6 Backup Set up... 7 Backup the

More information

Bootstrap guide for the File Station

Bootstrap guide for the File Station Bootstrap guide for the File Station Introduction Through the File Server it is possible to store files and create automated backups on a reliable, redundant storage system. NOTE: this guide considers

More information

Instructions for Accessing the Advanced Computing Facility Supercomputing Cluster at the University of Kansas

Instructions for Accessing the Advanced Computing Facility Supercomputing Cluster at the University of Kansas ACF Supercomputer Access Instructions 1 Instructions for Accessing the Advanced Computing Facility Supercomputing Cluster at the University of Kansas ACF Supercomputer Access Instructions 2 Contents Instructions

More information

MATLAB Distributed Computing Server System Administrator s Guide. R2013b

MATLAB Distributed Computing Server System Administrator s Guide. R2013b MATLAB Distributed Computing Server System Administrator s Guide R2013b How to Contact MathWorks www.mathworks.com Web comp.soft-sys.matlab Newsgroup www.mathworks.com/contact_ts.html Technical Support

More information

Martinos Center Compute Clusters

Martinos Center Compute Clusters Intro What are the compute clusters How to gain access Housekeeping Usage Log In Submitting Jobs Queues Request CPUs/vmem Email Status I/O Interactive Dependencies Daisy Chain Wrapper Script In Progress

More information

Tutorial: Using WestGrid. Drew Leske Compute Canada/WestGrid Site Lead University of Victoria

Tutorial: Using WestGrid. Drew Leske Compute Canada/WestGrid Site Lead University of Victoria Tutorial: Using WestGrid Drew Leske Compute Canada/WestGrid Site Lead University of Victoria Fall 2013 Seminar Series Date Speaker Topic 23 September Lindsay Sill Introduction to WestGrid 9 October Drew

More information

Backing Up TestTrack Native Project Databases

Backing Up TestTrack Native Project Databases Backing Up TestTrack Native Project Databases TestTrack projects should be backed up regularly. You can use the TestTrack Native Database Backup Command Line Utility to back up TestTrack 2012 and later

More information

bwgrid Treff MA/HD Sabine Richling, Heinz Kredel Universitätsrechenzentrum Heidelberg Rechenzentrum Universität Mannheim 29.

bwgrid Treff MA/HD Sabine Richling, Heinz Kredel Universitätsrechenzentrum Heidelberg Rechenzentrum Universität Mannheim 29. bwgrid Treff MA/HD Sabine Richling, Heinz Kredel Universitätsrechenzentrum Heidelberg Rechenzentrum Universität Mannheim 29. September 2010 Richling/Kredel (URZ/RUM) bwgrid Treff WS 2010/2011 1 / 25 Course

More information

PBS Tutorial. Fangrui Ma Universit of Nebraska-Lincoln. October 26th, 2007

PBS Tutorial. Fangrui Ma Universit of Nebraska-Lincoln. October 26th, 2007 PBS Tutorial Fangrui Ma Universit of Nebraska-Lincoln October 26th, 2007 Abstract In this tutorial we gave a brief introduction to using PBS Pro. We gave examples on how to write control script, and submit

More information

Data management on HPC platforms

Data management on HPC platforms Data management on HPC platforms Transferring data and handling code with Git scitas.epfl.ch September 10, 2015 http://bit.ly/1jkghz4 What kind of data Categorizing data to define a strategy Based on size?

More information

Installation and Configuration Guide for Windows and Linux

Installation and Configuration Guide for Windows and Linux Installation and Configuration Guide for Windows and Linux vcenter Operations Manager 5.0.3 This document supports the version of each product listed and supports all subsequent versions until the document

More information

The RWTH Compute Cluster Environment

The RWTH Compute Cluster Environment The RWTH Compute Cluster Environment Tim Cramer 11.03.2013 Source: D. Both, Bull GmbH Rechen- und Kommunikationszentrum (RZ) How to login Frontends cluster.rz.rwth-aachen.de cluster-x.rz.rwth-aachen.de

More information

Week Overview. Running Live Linux Sending email from command line scp and sftp utilities

Week Overview. Running Live Linux Sending email from command line scp and sftp utilities ULI101 Week 06a Week Overview Running Live Linux Sending email from command line scp and sftp utilities Live Linux Most major Linux distributions offer a Live version, which allows users to run the OS

More information

wu.cloud: Insights Gained from Operating a Private Cloud System

wu.cloud: Insights Gained from Operating a Private Cloud System wu.cloud: Insights Gained from Operating a Private Cloud System Stefan Theußl, Institute for Statistics and Mathematics WU Wirtschaftsuniversität Wien March 23, 2011 1 / 14 Introduction In statistics we

More information

CYCLOPE let s talk productivity

CYCLOPE let s talk productivity Cyclope 6 Installation Guide CYCLOPE let s talk productivity Cyclope Employee Surveillance Solution is provided by Cyclope Series 2003-2014 1 P age Table of Contents 1. Cyclope Employee Surveillance Solution

More information

ADAM 5.5. System Requirements

ADAM 5.5. System Requirements ADAM 5.5 System Requirements 1 1. Overview The schema below shows an overview of the ADAM components that will be installed and set up. ADAM Server: hosts the ADAM core components. You must install the

More information

Online Backup Client User Manual

Online Backup Client User Manual Online Backup Client User Manual Software version 3.21 For Linux distributions January 2011 Version 2.0 Disclaimer This document is compiled with the greatest possible care. However, errors might have

More information

IBM WebSphere Application Server Version 7.0

IBM WebSphere Application Server Version 7.0 IBM WebSphere Application Server Version 7.0 Centralized Installation Manager for IBM WebSphere Application Server Network Deployment Version 7.0 Note: Before using this information, be sure to read the

More information

Getting Started with HPC

Getting Started with HPC Getting Started with HPC An Introduction to the Minerva High Performance Computing Resource 17 Sep 2013 Outline of Topics Introduction HPC Accounts Logging onto the HPC Clusters Common Linux Commands Storage

More information

New High-performance computing cluster: PAULI. Sascha Frick Institute for Physical Chemistry

New High-performance computing cluster: PAULI. Sascha Frick Institute for Physical Chemistry New High-performance computing cluster: PAULI Sascha Frick Institute for Physical Chemistry 02/05/2012 Sascha Frick (PHC) HPC cluster pauli 02/05/2012 1 / 24 Outline 1 About this seminar 2 New Hardware

More information

NETWRIX EVENT LOG MANAGER

NETWRIX EVENT LOG MANAGER NETWRIX EVENT LOG MANAGER QUICK-START GUIDE FOR THE ENTERPRISE EDITION Product Version: 4.0 July/2012. Legal Notice The information in this publication is furnished for information use only, and does not

More information

Using SSH Secure FTP Client INFORMATION TECHNOLOGY SERVICES California State University, Los Angeles Version 2.0 Fall 2008.

Using SSH Secure FTP Client INFORMATION TECHNOLOGY SERVICES California State University, Los Angeles Version 2.0 Fall 2008. Using SSH Secure FTP Client INFORMATION TECHNOLOGY SERVICES California State University, Los Angeles Version 2.0 Fall 2008 Contents Starting SSH Secure FTP Client... 2 Exploring SSH Secure FTP Client...

More information

:Introducing Star-P. The Open Platform for Parallel Application Development. Yoel Jacobsen E&M Computing LTD yoel@emet.co.il

:Introducing Star-P. The Open Platform for Parallel Application Development. Yoel Jacobsen E&M Computing LTD yoel@emet.co.il :Introducing Star-P The Open Platform for Parallel Application Development Yoel Jacobsen E&M Computing LTD yoel@emet.co.il The case for VHLLs Functional / applicative / very high-level languages allow

More information

LOCKSS on LINUX. Installation Manual and the OpenBSD Transition 02/17/2011

LOCKSS on LINUX. Installation Manual and the OpenBSD Transition 02/17/2011 LOCKSS on LINUX Installation Manual and the OpenBSD Transition 02/17/2011 1 Table of Contents Overview... 3 LOCKSS Hardware... 5 Installation Checklist... 7 BIOS Settings... 10 Installation... 11 Firewall

More information

Installation and Configuration Guide for Windows and Linux

Installation and Configuration Guide for Windows and Linux Installation and Configuration Guide for Windows and Linux vcenter Operations Manager 5.7 This document supports the version of each product listed and supports all subsequent versions until the document

More information

HPC at IU Overview. Abhinav Thota Research Technologies Indiana University

HPC at IU Overview. Abhinav Thota Research Technologies Indiana University HPC at IU Overview Abhinav Thota Research Technologies Indiana University What is HPC/cyberinfrastructure? Why should you care? Data sizes are growing Need to get to the solution faster Compute power is

More information

Miami University RedHawk Cluster Connecting to the Cluster Using Windows

Miami University RedHawk Cluster Connecting to the Cluster Using Windows Miami University RedHawk Cluster Connecting to the Cluster Using Windows The RedHawk cluster is a general purpose research computing resource available to support the research community at Miami University.

More information

NTP Software QFS for NAS, NetApp Edition Installation Guide

NTP Software QFS for NAS, NetApp Edition Installation Guide NTP Software QFS for NAS, NetApp Edition Installation Guide This Installation Guide applies to all NTP Software QFS for NAS, NetApp filer editions. Abstract This guide provides a short introduction to

More information

Speeding up MATLAB and Simulink Applications

Speeding up MATLAB and Simulink Applications Speeding up MATLAB and Simulink Applications 2009 The MathWorks, Inc. Customer Tour 2009 Today s Schedule Introduction to Parallel Computing with MATLAB and Simulink Break Master Class on Speeding Up MATLAB

More information

Introduction to Running Computations on the High Performance Clusters at the Center for Computational Research

Introduction to Running Computations on the High Performance Clusters at the Center for Computational Research ! Introduction to Running Computations on the High Performance Clusters at the Center for Computational Research! Cynthia Cornelius! Center for Computational Research University at Buffalo, SUNY! cdc at

More information

PROGRAMMING FOR BIOLOGISTS. BIOL 6297 Monday, Wednesday 10 am -12 pm

PROGRAMMING FOR BIOLOGISTS. BIOL 6297 Monday, Wednesday 10 am -12 pm PROGRAMMING FOR BIOLOGISTS BIOL 6297 Monday, Wednesday 10 am -12 pm Tomorrow is Ada Lovelace Day Ada Lovelace was the first person to write a computer program Today s Lecture Overview of the course Philosophy

More information

Connecting to the School of Computing Servers and Transferring Files

Connecting to the School of Computing Servers and Transferring Files Connecting to the School of Computing Servers and Transferring Files Connecting This document will provide instructions on how to connect to the School of Computing s server. Connect Using a Mac or Linux

More information

Accessing VirtualBox Guests from Host using SSH, WinSCP and Tunnelling

Accessing VirtualBox Guests from Host using SSH, WinSCP and Tunnelling Accessing VirtualBox Guests from Host using S... 1 Accessing VirtualBox Guests from Host using SSH, WinSCP and Tunnelling By Steven Gordon on Thu, 15/08/2013-3:49pm Previously I described how to setup

More information

Server Installation Manual 4.4.1

Server Installation Manual 4.4.1 Server Installation Manual 4.4.1 1. Product Information Product: BackupAgent Server Version: 4.4.1 2. Introduction BackupAgent Server has several features. The application is a web application and offers:

More information

Batch Systems. provide a mechanism for submitting, launching, and tracking jobs on a shared resource

Batch Systems. provide a mechanism for submitting, launching, and tracking jobs on a shared resource PBS INTERNALS PBS & TORQUE PBS (Portable Batch System)-software system for managing system resources on workstations, SMP systems, MPPs and vector computers. It was based on Network Queuing System (NQS)

More information

BioHPC Web Computing Resources at CBSU

BioHPC Web Computing Resources at CBSU BioHPC Web Computing Resources at CBSU 3CPG workshop Robert Bukowski Computational Biology Service Unit http://cbsu.tc.cornell.edu/lab/doc/biohpc_web_tutorial.pdf BioHPC infrastructure at CBSU BioHPC Web

More information

HPC system startup manual (version 1.30)

HPC system startup manual (version 1.30) HPC system startup manual (version 1.30) Document change log Issue Date Change 1 12/1/2012 New document 2 10/22/2013 Added the information of supported OS 3 10/22/2013 Changed the example 1 for data download

More information

CS 3530 Operating Systems. L02 OS Intro Part 1 Dr. Ken Hoganson

CS 3530 Operating Systems. L02 OS Intro Part 1 Dr. Ken Hoganson CS 3530 Operating Systems L02 OS Intro Part 1 Dr. Ken Hoganson Chapter 1 Basic Concepts of Operating Systems Computer Systems A computer system consists of two basic types of components: Hardware components,

More information

McAfee Data Loss Prevention and later Virtual Appliance

McAfee Data Loss Prevention and later Virtual Appliance Installation Guide Revision B McAfee Data Loss Prevention 9.3.2 and later Virtual Appliance About McAfee Data Loss Prevention on virtual machines McAfee Data Loss Prevention (McAfee DLP) offers the option

More information

Using NeSI HPC Resources. NeSI Computational Science Team (support@nesi.org.nz)

Using NeSI HPC Resources. NeSI Computational Science Team (support@nesi.org.nz) NeSI Computational Science Team (support@nesi.org.nz) Outline 1 About Us About NeSI Our Facilities 2 Using the Cluster Suitable Work What to expect Parallel speedup Data Getting to the Login Node 3 Submitting

More information

Additional Information: SSH, PuTTY, and VmWare

Additional Information: SSH, PuTTY, and VmWare Additional Information: SSH, PuTTY, and VmWare Gianpierre Villagomez EECS 268 Additional Information: SSH, PuTTY, and VMware 1 EECS Linux machine remote access There are several ways you can access EECS

More information

LOCKSS on LINUX. CentOS6 Installation Manual 08/22/2013

LOCKSS on LINUX. CentOS6 Installation Manual 08/22/2013 LOCKSS on LINUX CentOS6 Installation Manual 08/22/2013 1 Table of Contents Overview... 3 LOCKSS Hardware... 5 Installation Checklist... 6 BIOS Settings... 9 Installation... 10 Firewall Configuration...

More information

Pearl Echo Installation Checklist

Pearl Echo Installation Checklist Pearl Echo Installation Checklist Use this checklist to enter critical installation and setup information that will be required to install Pearl Echo in your network. For detailed deployment instructions

More information

Clearswift Information Governance

Clearswift Information Governance Clearswift Information Governance Implementing the CLEARSWIFT SECURE Encryption Portal on the CLEARSWIFT SECURE Email Gateway Version 1.10 02/09/13 Contents 1 Introduction... 3 2 How it Works... 4 3 Configuration

More information

Grid Computing in SAS 9.4 Third Edition

Grid Computing in SAS 9.4 Third Edition Grid Computing in SAS 9.4 Third Edition SAS Documentation The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2014. Grid Computing in SAS 9.4, Third Edition. Cary, NC:

More information

Scyld Cloud Manager User Guide

Scyld Cloud Manager User Guide Scyld Cloud Manager User Guide Preface This guide describes how to use the Scyld Cloud Manager (SCM) web portal application. Contacting Penguin Computing 45800 Northport Loop West Fremont, CA 94538 1-888-PENGUIN

More information