Background. These efforts tend to be multi-institutional. Institutes and universities have varying infrastructure and schedulers installed

Transcription

1

2 Background Modern scientific efforts underway that require large scale parallel computations These efforts tend to be multi-institutional Institutes and universities have varying infrastructure and schedulers installed

3 Background (2) commonfund.nih.gov/hmp/ hmpdacc.org

4 The Problem Each organizaton ends up creating custom modules and code to talk to their scheduler/drm Sometimes this code has baked-in assumptions that make portability/sharing difficult Ends up slowing down the collaborative effort

5 Differences in Usage Sun Grid Engine Submissions mainly done with the qsub command Condor Submissions mainly done with a submission file and condor_submit -e # set the path to STDERR -o # Set the path to STDOT -q # Define which queue to send jobs to -P # Define what project assoc with job -p # Define the priority -wd # Define the working directory executable = /path/to/command universe = vanilla input = test.data output = loop.out error = loop.error log = loop.log initialdir = run_1 queue

6 DRMAA "Distributed Resource Management Application API" Several schedulers provide DMRAA support: SGE, Condor, LSF Basic Perl support via the Schedule::DRMAAc library by Tim Harsch at Lawrence Livermore National Labs

7 Straight Up DRMAA Perl ( $er, $diagnosis ) = drmaa_init( undef ); die drmaa_strerror( $err ). "\n". $diagnosis if $err; ( $err, $jt, $diagnosis ) = drmaa_allocate_job_template(); die drmaa_strerror( $err ). "\n". $diagnosis if $err; ( $err, $diagnosis ) = drmaa_set_attribute( $jt, $DRMAA_REMOTE_COMMAND, '/bin/sleep' ); die drmaa_strerror( $err ). "\n". $diagnosis if $err; ( $err, $diagnosis ) = drmaa_set_vector_attribute( $jt, $DRMAA_V_ARGV, [ "1" ] ); die drmaa_strerror( $err ). "\n". $diagnosis if $err; ( $err, $jobid, $diagnosis ) = drmaa_run_job( $jt );

8 The Goal Provide a simpler more approachable and readable API Add features to allow for easy formulation of jobs, iteration and command chaining Be Scheduler/DRM agnostic Allow the user to access their scheduler's native features if necessary

9 Making a request object my $req = Grid::Request->new(); # or my $req2 = Grid::Request->new( command => "/bin/tar", output => "/path/to/my/stdout.log", error => "/path/to/my/stderr.log", );

10 Adding switches & options $req = Grid::Request->new(); $req->command("/bin/ls"); $req->add_param("-la"); $req->add_param("/home/directory");

11 Job Requirements: hostname # Setting a memory requirement my $req = Grid::Request->new( command => "/my/big/bio/job", ); $req->hosts("machine1,machine2"); $req->submit_and_wait();

12 Job Requirements: O/S # Setting an operating system requirement my $req = Grid::Request->new( command => "/usr/bin/myjob", ); $req->opsys("linux"); $req->submit_and_wait();

13 Job Requirements: memory # Setting a memory requirement my $req = Grid::Request->new( command => "/my/big/bio/job", ); $req->memory("256g"); $req->submit_and_wait();

14 Job Submission $req = Grid::Request->new(); # Job configuration here # Submit the job and keep going $req->submit(); # Can subsequently block $req->wait_for_request(); # Or block until the job is done $req->submit_and_wait();

15 Singleton vs. Bulk Jobs Simple requests yield 1 job execution per request. However, most schedulers support "array" or "bulk" jobs, in which a submission can generate many thousands of jobs. Typically used to do a parametric sweep of some argument.

16 Iterating over a directory # Scan a directory and use the file paths as arguments $req->add_dir_param('--country=$(name)', "/path/to/dir"); /path/to/dir usa.txt uk.txt brazil.txt japan.txt france.txt italy.txt --country=/path/to/dir/usa.txt --country=/path/to/dir/uk.txt --country=/path/to/dir/brazil.txt --country=/path/to/dir/france.txt --country=/path/to/dir/italy.txt

17 Iterating over a file # Scan a directory and use the lines as arguments $req->add_file_param('--city=$(name)', "/path/to/cities.txt"); /path/to/cities.txt --city=portland Portland --city=newyork NewYork --city=tokyo London Tokyo --city=paris Paris --city=rome Rome

18 Iterating over an array # Use array contents as arguments $req->add_array_param('-dwarf=$(name)', -dwarf=thorin thorin -dwarf=balin balin -dwarf=gloin gloin -dwarf=dwalin dwalin -dwarf=bofur bofur -dwarf=bombur bombur

19 Combining param types $req->add_file_param('--line=$(name)', "/path/to/file"); $req->add_dir_param('--file=$(name)', "/path/to/dir"); $req->add_array_param('--ele=$(name)', = $req->submit_and_wait(); Note: Need to be careful about the number of items in each param. They need to be the same. If they are not G::R will truncate lists to the lowest number across all parameters.

20 Block size Bulk jobs are bundled for execution on the grid. The list of jobs to execute is divided by the "block" size so that each grid jobs execute <block> number of iterations. The default block size is 100. Example # File has 12,000 lines $req->add_file_param('--line=$(name)', "/path/to/file"); # 120 grid jobs, each invoking the executable 100 times $req->submit();

21 Changing block size Scheduler configurations sometimes impose limits on runtime For some types of jobs, a block size of 100 is too much Larger block sizes can reduce scheduling overhead # Turn off bundling by setting the block size to 1 # Caution: may result in a very large number of grid jobs. $req->block_size(1); # Increase the block size to beyond the default # Caution: may result in very long runtimes. $req->block_size(1000);

22 Dynamic block size Sometimes, the right block size just depends on the circumstances Pass a code reference to the block_size() method Code reference must return a positive integer $req->block_size( sub { my ($com_obj, $count) my $block_size; if ($phase_of_moon eq "full") { $block_size = 1000; } else { $block_size = 50; } return $block_size; } );

23 Chaining Commands Possible to build a pipeline of commands with use of the new_command() method Simply moves a cursor of sorts A B Singleton jobs "Bulk" jobs C D

24 Chaining Commands (2) $req = Grid::Request->new(); # Create/setup first command $req->command("/bin/do/computation"); $req->new_command(); # Create/setup second command $req->command("/bin/post_process_results"); $req->submit();

25 Where to go from here Schedule::DRMAAc is built and installed for a single scheduler, so difficult to use G::R for multiple grids Possibly create an auxiliary, or new, module to save progress of a pipeline and allow for resumption at the last point of failure. Eliminate the requirement for a config file: use Grid::Request qw(sge);

26 Global Grids Several global grids out there to integrate with XSEDE (Successor to TeraGrid) xsede.org Open Science Grid opensciencegrid.org

27 Thank you