GRMS Features and Benefits

Transcription

1 GRMS - The resource management system for Clusterix computational environment Bogdan Ludwiczak [email protected] Poznań Supercomputing and Networking Center

2 Outline: GRMS - what it is? GRMS features What GRMS offers to its users? GRMS usage How they can use it? GRMS architecture

3 What GRMS is? GRMS is an open source scheduling system for large scale distributed computing infrastructures Designed to deal with resource management challenges in Grid environments: load-balancing among clusters, setting up execution environments before and after job execution, remote job submission and controlling, files staging, More. Based on the dynamic resource selection, mapping and advanced grid scheduling methodologies, combined with feedback control architecture.

4 GRMS Features (1) Job submission & control (cancel, suspend, resume) GRMS main objective: GRMS performs remote jobs control and management in the way that satisfies Users (Job Owners) and their applications requirements. Simultaneously, Resource Administrators (Resource Owners) have full control over resources on which all jobs and operations will be performed by appropriate GRMS setup and installation

5 GRMS Features (2) Choosing the best resource for the Job execution, according to Job Description and chosen mapping algorithm multicriteria algorithm Submitting the GRMS Job according to provided Job Description to chosen resource Job migration Job checkpointing: using defined checkpoint interface implemented by application

6 GRMS Features (3) Complex information about submitted jobs List of jobs submitted by user Information about the Job status Job Description used for submission Information about request progress Name of host where the Job is running Submission time Start time on resource Finish time History of job execution (migrations)

7 GRMS Features (4) Dynamic resource discovery Using multiple information sources about Grid environment standard Globus MDS (GIIS/GRIS infrastructure) Clusterix Infrastructure Monitoring System igrid/delphoi/mercury GridLab Services Support for file staging transferring input and output files and whole directories (GridFTP, GASS, Data Management Systems) Mechanism for registering for events notification Notifying about status changes and request progress (e.g. via or sms)

8 GRMS Features (5) Time constraints for running jobs Dynamically extending (during application runtime) job description by adding output data Support for workflow jobs: job can consist of set of independent tasks with or without precednence constraints

9 GRMS features developed in Clusterix Support for distributed MPICH-G2 application allows users to submit jobs which will be dispersed among many nodes of many clusters, makes Clusterix able to execute large, multi process application Prediction of Job execution Increases the resource management effectiveness by providing estimated values Allows resource broker to find out: job execution time, job pending time in given queue, probable resource utilization by the job estimation inaccuracy

10 Architecture

11 GRMS Interface GSI-enabled web service for all: clients applications middleware All users requirements are expressed through XML-based resource specification documents, called GRMS Job Description GRMS asigns a GRMS_JOB_ID to every submited job

12 Job Description(1) XML based language Job executable file location arguments file argument (files which have to be present in working directory of running executable) environment variables standard input standard output standard error checkpoint files

13 Job Description(2) Resource requirements of executable Name of host for job execution (if provided no scheduling algorithm is used) Operating system Required local resource management system Minimum memory required Minimum number of cpus required Minimum speed of cpu Network parameters: latency, bandwidth, capacity

14 <grmsjob appid="testjob" persistent="false"> </grmsjob> <simplejob> </simplejob> Job Description - example <resource> <osname> Linux </osname> <memory>128</memory> <cpucount>2</cpucount> </resource> <executable type="single" count="1"> </executable> <file name="exec-file" type="in"> </file> <arguments> </arguments> <stdin> </stdin> <stdout> </stdout> <url>file:////bin/grep</url> <value>pcss</value> <url>gsiftp://access/~/examples/datafile</url> <url>gsiftp://access/~/gclient/var/pbs.stdout.${job_id}</url

15 <grmsjob appid="mpichg2test" persistent="true"> Job Description MPICH-G2 example <simplejob> <resource> <hostname>access.pcss.clusterix.pl</hostname> <localrmname>pbs</localrmname> </reource> <executable type="mpichg" count="3"> <file name="clx" type="in"> <url>file:////tmp/clx_ia64_g2</url> </file> <arguments> <value>home/clx/var/grms_demo1v2</value> <value>25</value> <value>1</value> </arguments> <stdout> <url>gsiftp://access.pcss.clusterix.pl/~/demo1.out</url> </stdout> </executable> </simplejob> </grmsjob>

16 <grmsjob appid="mpichg2test" persistent="true"> <simplejob> <resource> <hostname tilesize="2">access.pcss.clusterix.pl</hostname> <localrmname>pbs</localrmname> </resource> <resource> <hostname tilesize="1">access.pcz.clusterix.pl</hostname> <localrmname>pbs</localrmname> </resource> <executable type="mpichg" count="3"> <file name="clx" type="in"> <url>file:////tmp/clx_ia64_g2</url> </file> <arguments> <value>home/clx/var/grms_demo2</value> <value>25</value> <value>1</value> </arguments> <stdout> <url>gsiftp://access.pcss.clusterix.pl/~/demo2.out</url> </stdout> </executable> </simplejob> </grmsjob>

17 Client applications Command line tool Web Portal Mobile Client

18 Client command line tool

19

20

21 Client - portal

22

23

24

25 Client mobile devices

26 GRMS deployments Polish projects: Clusterix national Linux based Grid SGI Grid Progress Project Other HPC-Europa InteliGrid GriPhyN Canadian Grid Initiative testing Swiss Grid, DEISA, TeraGrid being negotiated/tested

27 More information Admin Guide how to install GRMS verables/d9.6.admins_guide.pdf User Guide how to use GRMS ers_guide.pdf Source Code Branch without workflow support (1.x) Final version of GRMS (ver. 2.x)