Cloud Computing. Lectures 3 and 4 Grid Schedulers: Condor

Transcription

1 Cloud Computing Lectures 3 and 4 Grid Schedulers: Condor

2 Up until now Introduction. Definition of Cloud Computing. Grid Computing: Schedulers: Condor architecture.

3 Summary Condor: user perspective. Condor Flocking.

4 Job Submission Universe = standard input = program.in output = program.out executable = program Create a sub file: queue 3 % vi program.sub Submit the job: % condor_submit program.sub

5 Job Submission Executable = /bin/foo Arguments = xpto $(Process) Requirements = Memory >= 1024 && OpSys=="WINNT51" && Arch =="INTEL" Universe = vanilla input = test.data output = $(Process).out error = $(Process).error log = $(Process).log Initialdir = run_1 Queue 5 Initialdir = run_2 Queue 5

6 Job Submission Arch, OpSys, Disk (KB), Memory (MB), Machine, More: _Job.html

7 ClassAds ClassAds are Condor s mechanism for: Representing resources and clients within the system. Expressing client and machine preferences. Allocating resources. Sufficiently expressive for representing characteristics (features), requests and policies. Simple enough to allow matching (at the negotiator) between clients and resources. Can be listed using condor_status.

8 Condor_status example

9 ClassAds MyType = Machine TargetType = Job Machine = n3.grid.com Arch = INTEL OpSys = Linux Disk = Rank = (Customer==john?0:1) MyType = Job TargetType = Machine Owner = john Cmd = /usr/bin/java Rank = Kflops * 10 + Disk

10 Condor Scheduling Calculate the total available resources. Order requests by their users priority (lower is better). Priority starts with a configured value and decays with resource use for fairness. Calculate the proportional resource share by user priority. Start the jobs from the user with highest priority by order of machine preference followed by job preference. Continue with the next user.

11 Condor Applications Unix or Windows binary executables. Scripts. Interpreted programs (JVM, Mono, perl). MPI. PVM.

12 Universe Types Condor provides different universes: vanilla UNIX jobs + no Remote I/O. standard UNIX jobs + Remote I/O. scheduler UNIX jobs with immediate local execution. globus UNIX jobs over Globus. java Java apps. Finds and benchmarks the VM. parallel MPI jobs. Reserves nodes before starting job. vm Run a job inside a system virtual machine (VMWare or Xen).

13 vanilla Universe Allows users to submit any UNIX process to Condor. Pros: No program modification. Very flexible. Includes: Binaries. Scripts. Interpreted programs (java, perl). Multi-process jobs.

14 vanilla Universe (cont.) Cons: No checkpointing. Limited I/O at remote machines: Explicit description of input files. Explicit descriptions of output files. Condor does not start vanilla jobs at an unfriendly node. ClassAds: FilesystemDomain and UIDDomain must match.

15 When one connects clusters HELP! SOS! Cluster Cluster Cluster File Server File Server SOS! Cluster Cluster HELP! SOS! File Server File Server File Server File Server

16 Unfriendly Environments An executable may run with: Correct OS and HW architecture and enough memory. But some elements may be missing: Input files. Disk space for output files. Absence of shared file system. No login. Run as nobody?

17 standard Universe Allows users to submit jobs with special Condor relinking. Pros: Checkpointing Remote I/O: Friendly environment anywhere. Data buffering. I/O performance monitoring and reporting. Remapping of file names.

18 standard Universe (cont.) Cons: Applications must be relinked. Limited set of applications: Only single process UNIX apps. Certain system calls are restricted.

19 Restrictions on System Calls standard universe does not allow: Multiple processes: fork(), exec(), system() Inter-process communication : Semaphores, message passing, shared memory. Sophisticated I/O: mmap(), select(), poll(), non-blocking I/O, file locking. Threads.

20 Remote I/O Starter!!! file_remaps = "data =

21 Brief I/O Summary % condor_q -io -- Schedd: c01.cs.wisc.edu : < :2016> ID OWNER READ WRITE SEEK XPUT BUFSIZE BLKSIZE joe KB KB KB/s KB 32.0 KB joe KB KB B /s KB 32.0 KB joe 44.7 KB 22.1 KB B /s KB 32.0 KB 3 jobs; 0 idle, 3 running, 0 held Great for performance debugging!

22 Complete I/O Summary in Your condor job "/usr/joe/records.remote input output" exited with status 0. Total I/O: KB/s effective throughput 5 files opened 104 reads totaling KB 316 writes totaling 1.2 MB 102 seeks I/O by File: buffered file /usr/joe/output opened 2 times 4 reads totaling 12.4 KB 4 writes totaling 12.4 KB buffered file /usr/joe/input opened 2 times 100 reads totaling KB 311 write totaling 1.2 MB 101 seeks

23 File Remapping Suppose a program opens a file called data, but one wants to open a different file according to the process number. In the jobs sub file, add: file_remaps = "data = /home/john/data.$(process)" Process 1 gets /home/john/data.1 Process 2 gets /home/john/data.2 And so on And of course free access to distributed file systems.

24 Relinking Use condor_compile before usual compilation commands: For example: gcc main.o utils.o -o program Becomes: condor_compile gcc main.o utils.o -o program Despite the name (compile), it s just relinking with Condor libraries.

25 Checkpoint To checkpoint an executing program is to take a snapshot of its current state in such a way that the program can be restarted from that state at a later time possibly at a different resource. Provides: Preemption - Resume scheduling. Fault Tolerance when checkpointing is done periodically. In Condor, checkpointing running jobs is optional. If it is needed, source should be linked with condor_syscall_lib.

26 Checkpointing in Condor Implemented in condor_syscall_lib as a signal handler When condor sends a signal to checkpoint, the handler saves process state information in a checkpoint file From Core - contents of process uarea, data and stack segments From Executable symbol and debugging info, initialized data, text

27 Checkpointing & Restart Shadow sends the latest checkpoint file to the new Starter during restart The starter, reads the job state from the checkpoint file and the execution continues Starter periodically sends a checkpoint signal to the executing job Condor_syscall_lib makes job dump core and saves job state in the checkpoint file Checkpoint file temporarily Remote Machine Starter transfers latest checkpoint file to shadow when job vacated Checkpoint signal Starter process for the remote job Checkpoint file Code in condor_syscall_lib saves process state information Checkpoint file transferred when job vacated Checkpoint file transferred when job restarted Local File System Shadow process for the job Remote Machine Submit Machine

28 Ganglia: GUI for Grid Monitoring

29 DAGMan Directed Acyclic Graph Manager Manages dependencies between processes: Don t run B before A finishes. The execution plan is represented as a directed acyclical graph (DAG), where: Nodes are jobs. Edges are dependencies.

30 Defining DAGs A DAG is specified in a.dag file that lists the tasks and their dependencies. For example: # diamond.dag Job A a.sub Job B b.sub Job C c.sub Job D d.sub Parent A Child B C Parent B C Child D Job B Job A Job D Each node corresponds to the job described in its.sub file. Job C

31 Running a DAG % condor_submit_dag diamond.dag Starts a daemon process to follow the execution and interact with the schedd. It s a meta-scheduler: controls the scheduler. Only submits jobs when the plan allows for it. Processing the DAG results in a list of execution levels. Level 1 A Level 2 B C D Level 3 E

32 DAG: other features Associate scripts to jobs: SCRIPT PRE e SCRIPT POST Rescue: If a job fails, DAGMan generates a.dag.rescue file with the missing part of the DAG. Retry: If a job fails, it may be reexecuted: RETRY A 5 Throttling: It is possible to limit the number of concurrent jobs: condor_submit_dag maxjobs N

33 Condor: Flocking It s a compilation configuration + configuration file describing the other pools. Gateways share job and node characteristics among themselves.

34 Globus. Next time