Using Globus Toolkit

Transcription

1 Using Globus Toolkit G. Poghosyan & D. Nilsen GridKa School September 2006

2 Basic Grid Services in GT Security Services GSI (Grid Security Infrastructure) Data Services GridFTP RFT (Reliable File Transfer) RLS (Replica Location Service) Execution Management Services GRAM (Grid Resource Allocation and Management) Information Services MDS (Monitoring and Discovery System) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 2

3 Grid Security Infrastructure (GSI) Based on Public key cryptography X.509 certificate (grid-cert-request) Very sensitive to system clock skew use NTP Provides Mutual authentication Single sign-on Grid Security Infrastructure (GSI) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 3

4 Certificate/Proxy - Single Sign-on - Athorization - Job Finding information on a certificate grid-cert-info subject -startdate -issuer -enddate -file certfile Finding information on a proxy grid-proxy-info [-exists] -debug Creating a proxy grid-proxy-init verify -q -valid h:m -debug Default value is valid 12:00 Destroying a proxy grid-proxy-destroy -dryrun [file1 ] Grid Security Infrastructure (GSI) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 4

5 Certificate/Proxy - Single Sign-on Gives information about the proxy $ grid-proxy-info Create proxy at /tmp/x509up_u<user ID> $ grid-proxy-init -debug -verify Gives information which proxy will be destroyed $ grid-proxy-destroy -dryrun Destroy proxy certificate $ grid-proxy-destroy Grid Security Infrastructure (GSI) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 5

6 Mutual Authentication - MyProxy - Athorization - Job Storing a credential in a repository myproxy-init [-s hostname] [-c credlifehours] [ t proxylifehours] Default values are c 168 (7 days) and t 12 Display info on stored credentials myproxy-info [-s hostname] Retrieving a credential myproxy-get-delegation (myproxy-logon) [-s hostname] [-t hours] Removing a credential myproxy-destroy [-s hostname] Grid Security Infrastructure (GSI) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 6

7 Mutual Authentication - MyProxy Send myproxy certificate to MyProxy-Server $ myproxy-init -s xen-14-2.fzk.de Getting myproxy certificate $ myproxy-get-delegation -s xen-14-2.fzk.de Destroy proxy certificate $ myproxy-destroy -s xen-14-2.fzk.de Set enviroment export MYPROXY_SERVER= xen-14-2.fzk.de Grid Security Infrastructure (GSI) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 7

8 Data Services Handle movement, access, replication and updates of data Example services in Globus Toolkit GridFTP Reliable File Transfer (RFT) Replica Location Service (RLS) Data Services (GridFTP) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 8

9 Data Services - GridFTP Third Party FTP server Basic tool provided with Globus Toolkit globus-url-copy [-tcp-bs TCPBufSizeBytes] [-p parallelism] sourceurl desturl URL format = protocol://[host]:[port]/path Supported protocols: gsiftp://, file://(on a local machine only), ftp://, It is legal to specify a user name and password in the the URL as follows gsiftp://myname:[mypassword]@myhost.mydomain.com/file.dat Data Services (GridFTP) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 9

10 Data Services - GridFTP local local $ globus-url-copy gsiftp://localhost/etc/hosts file:///tmp/hosts_copy remote local $ globus-url-copy gsiftp://iwrcggt4.fzk.de/etc/hosts file:///tmp/hosts_copy_<ihrname> local remote $ globus-url-copy file:///tmp/hosts_copy_<ihrname> gsiftp://iwrcggt4.fzk.de/tmp Data Services (GridFTP) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 10

11 Data Services - Reliable File Transfer (RFT) Manages a set of third-party GridFTP transfers Uses a database to checkpoint transfer state Recovers from Source/destination server failures Network failures Container failures Transfers retried with exponential backoff Resumes transfers where they left off Gevorg Poghosyan and Dimitri Nilsen Data Services (RFT) GridKa School, September 2006, Karlsruhe, Germany 11

12 Data Services - Reliable File Transfer (RFT) Simply pass it a file with options at the top, and then source / destination URL pairs. rft f my.xfr rft-delete f filename Other options -h host (defaults to localhost) -r port (defaults to 8080) -l lifetime (default 60 minutes) -m security (msg, conv, trans (default)) -p protection (sig, enc (default)) -z authorization (self, host (default)) -file file to write epr to Data Services (RFT) Gevorg Poghosyan and Dimitri Nilsen GridKa School, September 2006, Karlsruhe, Germany 12

13 Data Services - Reliable File Transfer (RFT) my.xfr #true=binary mode false=ascii true #Block size in bytes #TCP Buffer size in bytes #No thirdpartytransfer (Notpt) false #Number of parallel streams 1 #Data Channel Authentication true Gevorg Poghosyan and Dimitri Nilsen # Concurrency of the request 1 #Grid Subject name of the source gridftp server null #Grid Subject name - destination gridftp server null #Transfer all or none of the transfers false #Maximum number of retries 10 #Source/Dest URL Pairs gsiftp://iwrcggt4.fzk.de/etc/group gsiftp://iwrcggt4.fzk.de/tmp/deleteme Data Services (RFT) GridKa School, September 2006, Karlsruhe, Germany 13

14 Data Services - Reliable File Transfer (RFT) Get.xrf file $ cp $GLOBUS_LOCATION/share/globus_wsrf_rft_test/ \ transfer.xfr /tmp Edit.xfr file and create files to transfer $ vi /tmp/transfer.xfr $ touch /tmp/rfttest.tmp RFT-Test $ rft -h <FQDN> -f /tmp/transfer.xfr Grid Security Infrastructure (GSI) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 14

15 Data Services - Replica Location Service (RLS) Co-developed by Globus and DataGrid projects Provides a mechanism for registering and locating replicas Maintains mappings between logical names and target names Logical names unique identifiers for data content that may have one or more physical replicas Target name physical location of a replica or other logical names representing the data Gevorg Poghosyan and Dimitri Nilsen Data Services (RLS) GridKa School, September 2006, Karlsruhe, Germany 15

16 Execution Management Services Handles placement, provisioning and lifetime management of jobs Example services in Globus Toolkit Grid Resource Allocation and Management (GRAM) Provides a single standard interface for remote job submission and resource management Requires users and application developers to learn only one method to gain access to a large variety of local management systems Gevorg Poghosyan and Dimitri Nilsen Execution Management (GRAM) GridKa School, September 2006, Karlsruhe, Germany 16

17 Pre-WS GRAM (GT2) Command Line Tools Interactive job submission globus-job-run host:port/jobmanager-pbs command Batch job submission globus-job-submit command Checking job status when done output globus-job-status jobid globus-job-output out err jobid Kill a job globus-job-cancel jobid Gevorg Poghosyan and Dimitri Nilsen Execution Management (Pre-WS-GRAM) GridKa School, September 2006, Karlsruhe, Germany 17

18 WS GRAM (GT4) Command Line Tools Interactive job submission globusrun-ws submit F JobFactoryURL Ft FactoryType s q c command Batch job submission globusrun-ws submit batch F JobFactoryURL Ft FactoryType o EPRfile c command Checking job status globusrun-ws status j EPRfile Kill a job globusrun-ws kill j EPRfile Gevorg Poghosyan and Dimitri Nilsen Execution Management (WS-GRAM) GridKa School, September 2006, Karlsruhe, Germany 18

19 WS-GRAM WS oriented job submission command(gt4) $ globusrun-ws -submit -c /bin/hostname Streaming the output $ globusrun-ws -submit -s -c /bin/hostname output go to file $ globusrun-ws -submit -s -so out.txt -c /bin/hostname Send job to remote host $ globusrun-ws -submit -s -F <host> /bin/hostname Send to the batch system (pbs) $ globusrun-ws -submit -s -F iwrcggt4.fzk.de -Ft PBS -c /bin/hostname Grid Security Infrastructure (GSI) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 19

20 WS-GRAM: End point reference End point reference globusrun-ws -submit -batch -c /bin/hostname createa batch job $ vi batchjob.sh #!/bin/bash sleep 10 $chmod +x batchjob create an end point ree file ferenc $ globusrun-ws -submit -batch -o epr1.file \ -c $HOME/batchJob.sh get status $ globusrun-ws -status -j epr1.file Grid Security Infrastructure (GSI) G.Poghosyan&D.Nilsen GridKa School, September 2006, Karlsruhe, Germany 20

21 WS-GRAM Client Job functions Delegate GT4 Java Container Delegation GRAM services RFT File Transfer Transfer request Job events Local job control Delegate FTP control sudo SEG GRAM adapter GridFTP Compute element Local scheduler User job GridFTP FTP data Remote storage element(s) Gevorg Poghosyan and Dimitri Nilsen Execution Management (WS-GRAM) GridKa School, September 2006, Karlsruhe, Germany 21

22 Information Services Provide access to information about applications, resources and services in a grid environment Example services in Globus Toolkit Monitoring and Discovery System (MDS) Used to monitor and discover resources and services that exist in a grid environment MDS2 Pre-Web Service version of MDS (GT2) MDS4 Web Service based version of MDS (GT4) Gevorg Poghosyan and Dimitri Nilsen Information Services (MDS) GridKa School, September 2006, Karlsruhe, Germany 22

23 MDS2 Uses Lightweight Directory Access Protocol (LDAP) to access this information Grid Resource Information Service (GRIS) Provides resource information Grid Index Information Service (GIIS) Provides caching services and search capabilities GRIS can register with a GIIS GIIS can register with other GIIS to create hierarchy Gevorg Poghosyan and Dimitri Nilsen Information Services (MDS2) GridKa School, September 2006, Karlsruhe, Germany 23

24 MDS4 (WS MDS) Includes two WSRF-based services Index Service Collects data and provides a query/subscription interface to the data Can create hierarchy of index services Trigger Service Collects data and takes actions based on the data wsrf-query s e '/* mds-servicegroup-add [options] config.xml Gevorg Poghosyan and Dimitri Nilsen Information Services (MDS4) GridKa School, September 2006, Karlsruhe, Germany 24

25 Good Luck in looking-for useful commands and services in Information Services (MDS4)