Condor: Grid Scheduler and the Cloud

Transcription

1 Condor: Grid Scheduler and the Cloud Matthew Farrellee Senior Software Engineer, Red Hat 1

2 Agenda What is Condor Architecture Condor s ClassAd Language Common Use Cases Virtual Machine management Cloud aggregation/bridging Condor and building clouds 2

3 What is Condor Open source project out of the University of Wisconsin-Madison, Distributed computing research project in Computer Sciences, est To many a batch system managing millions of machines worldwide, running many more jobs for individuals, enterprises, governments and research organizations Under active development by multiple organizations Maintaining an active user community Multi-platform code base RHEL, HPUX, AIX, SLES, YDL, Solaris, Debian, FreeBSD, OS X, Windows IA64, x86, X86_64, HPPA, PPC64, Cell, UltraSPARC Distributed by multiple organizations, e.g. UW and Red Hat in Fedora and RHEL (MRG) A fundamental building block for creating clouds 3

4 Architecture Submit nodes Manage a queue of jobs Responsible for all aspects of a job s life Can be highly available, any one in an activepassive set Central Manager Rendezvous point Global scheduling/matchmaking responsibilities Can be highly available, in an active-active set Execution nodes The workers, responsible for job execution Have their own policies Submit Node Central Manager Execute Node Execute Node Execute Node Execute Node 4

5 Condor s ClassAd Language Metadata language to describe everything in Condor Jobs (job ad), Execution resources (machine/slot ad), Users (submitter ad), Daemons (collector ad, schedd ad, etc) Schema-free set of name-value pairs [MyType = Job, Requirements = Memory > 1024, Owner = matt, ] [MyType = Machine, Memory = 4096, KFlops = , LoadAvg = 0.04, Requirements = Owner == matt && ImageSize < Memory, ] Values are typed numbers, strings, expressions, undefined The expression type enables policy and control Expressions are Boolean logic, including string processing functions Expressions are evaluated in a context 5

6 Condor s ClassAd Language: Expressions Scheduling Job require machines: Requirements = Memory > 1GB && KFlops > && OpSys == Linux && ForCloud =?= TRUE Job rank machines: Rank = Memory Machine require jobs: START = (LoadAvg CondorLoadAvg) <= 0.3 && (Memory * 0.8) > ImageSize && stringlistmember(owner, john,jane ) Machine rank jobs: RANK = (Owner == john ) * 10 + (Owner == jane ) Admin policy: PreemptionRequirements = Owner =!= jimmy Job policy OnExitRemove = (ExitBySignal == False) && (ExitCode == 0) PeriodicHold = ImageSize > 4GB Machine policy PREEMPT = ImageSize > (Memory * 2) 6

7 Common Use Cases Batch processing Virtual machine management Applying policy Cloud aggregation/bridging 7

8 Batch processing Millions of things to do (jobs), thousands of machines to do them on Condor manages the machines transfers data runs the jobs schedules between users enacts policy maintains accounting information Condor overhead is low. No need for millions of jobs to amortize, dozens will do, and benefits of policy, notification, persistence still apply 8

9 Virtual Machine management The job is the virtual machine Condor manages the virtual machine s life-cycle, gathers accounting data, implements policy, transfers data, etc. Condor manages the VM via libvirt ( or VMware Xen and KVM accessible via libvirt Use cases Tightly controlled environment, inside VM Present Windows environment on Linux or vice versa Provide checkpointing where otherwise could not VM sandboxing, run programs requiring privileges Co-locate multiple user s VMs (multi-tenancy) 9

10 Applying policy Condor provides useful default policies, e.g. VMs run to completion, notify user of results/usage Machines if load is high, don t run new VMs Global don t interfere with user or machine requirements When a Condor pool has complex, demanding users, or resources provided by different people/organizations, customized policies my be in order, e.g. VMs run multiple concurrent instances, start on Black Friday or semi-monthly, re-run after fault Machines only run VMs from owner s group between 9 and 5, everyone else has a low priority shot from 5 to 9 Global control limiters (e.g. NFS mount users, licenses), distribute work to already busy machines (packing) 10

11 Cloud aggregation/bridging Condor has the ability to send VMs to other resource managers Condor becomes the unified interface to many types of resources internal VM resources and multiple external clouds Condor s life-cycle management, accounting and policy benefits still available Use cases Manage overflow/spillover Access to specialized resource managers Transformation between VM types/systems Allow a single app/stack to bridge multiple clouds 11

12 Cloud aggregation/bridging: Architecture = Process Spawned = Communication Submit Node schedd job router gridmanager Schedd accepts jobs over SOAP, QMF ( CLI a GAHP Grid ASCII Helper Protocol An adapter to an external resource manager Exist for many batch systems Exists for EC2-like resource managers Extensible to new resource managers Job Router transforms types, e.g. stack to VM to EC2 AMI ec2-gahp Amazon EC2 *-gahp *, e.g. Condor, Globus, 12

13 Condor and building Clouds Condor IaaS cloud VM manager capabilities Numerous deployments already Resource accounting present Flexible policy for SLA VM environment management via libvirt, e.g. network and storage Cross cloud aggregation/orchestration Security Authentication (e.g. SSL, ), Integrity, Encryption Condor PaaS cloud job/vm manager capabilities Success enabling concurrent Hadoop ( workloads Flexibility in policy and ClassAds allows for platform integration, with and around Condor 13

14 Cloud Example A private cloud serving R&D virtual machines with month long interactive logins, exposing Red Hat Enterprise Linux and Windows workstations Condor is the core VM management/scheduling component Condor with high availability central managers and schedds Partitionable slots dynamically carve up resources to pack in VMs Concurrency limits help limit licensed VM images Customized portal UI over Condor APIs, e.g. WSDL Billing/charge-back over Condor accounting/logging functionality Additionally, managed backup, monitoring, migration, help-desk, 14

15 Q & A 15

16 Abstract Condor is an open source project from the University of Wisconsin-Madison. With deep roots as a distributed batch scheduling system, it has expanded over the past decade as a manager of resource managers, and over the past several years as a manager of virtual machines. This talk will present the basic architecture of a Condor pool, describe and discuss Condor s lingua franca, ClassAds, then dive into virtual machine management for building clouds and integration with existing clouds, such as EC2. 16

17 Architecture: Systems = Process Spawned = ClassAd Communication Pathway Submit-Only master schedd schedd Central Manager startd Regular Node master startd schedd master negotiator collector Regular Node master startd schedd Execute-Only master startd Execute-Only master startd 17

18 Architecture: Systems - Components Master monitor/manage Condor daemons on a machine, e.g. start, stop, restart on failure Schedd job manager life-cycle management, policy implementation, job data, user/job accounting Collector directory service of ClassAd data, e.g. lookup service for daemons and user tools Negotiator scheduler matching jobs with machines, implementing policy, user/machine accounting Startd represent machine, execute jobs, implements policy 18

19 Use Case: Workflow processing When there is structure to the work, e.g. dependencies Condor provides DAGMan (directed acyclic graph manager) An application on top of Condor Uses Condor interfaces to monitor/submit Submits based on dependencies A DAG is a graph of nodes, each of which is a job/vm Scales to 100Ks of nodes Tolerant to faults A B D C B and C must wait for A D must wait for B and C Nodes can run programs, stage data, monitor itself 19