INRIA Job Scheduler GE webinar WP4 Cloud Hosting Chapter Lorenzo CANTELMI, INRIA lorenzo.cantelmi@inria.fr
Main requirements across Computing Systems eras Batch Systems Personal Computers Grids & Clusters (& Middleware) Cloud Computing (& Middleware) Programma 101 Time is passing, but some concepts remain still the same: - job -----------------------------------------------------------------------> portability - resource -------------------------------------------------------------> homogeneity & management - scheduler(resources, jobs) ---------------------------------------> productivity time
Main Motivations and details about implementation Most of enterprises are not partially/totally cloudified yet, so they rely still on physical hardware (workstations, hosts, and so on) If those enterprises decided to move totally in the cloud, some physical resources could be at their disposal, but underutilized (for instance, a good workstation is too much for a single developer) Job Scheduler GE allows to execute a generic job (XML), by involving an heterogeneous type of resources, both physical and virtual (i.e. VMs). How? Job Scheduler GE leverages the open-source middle-ware. ProActive OK, what is a middle-ware? The glue which connects objects which are distributed across multiple heterogeneous computer systems So, by enabling each resource, both physical and virtual, to be part of the middle-ware, Job Scheduler GE can increase resources usage and job portability. It might act as a flexible PaaS or orchestrator. Enabled <=> a JVM with ProActive lib is registered and running Cloud computing does not address to fill such a hybrid business model.
Global Architecture Working on REST Server Being a middle-ware, the architecture could be apparently complex. Getting it easier: octopus-style! Scheduler: brain Dataspace: reminds - if any! :) Resource Manager: set of tentacles Node: tentacle Message Queuing System: nerves, that allow communications between all of them to take place. REST is other ways to drive the monster, other than JAVA API and CLI. Portal and Workflow Studio GUI might be integrated, as well.
Job definition Submit a job to the Job Scheduler means submitting a task-flow, which is a set of tasks. A task is the smallest schedulable entity Tasks well-designed <=> Job well-done Two types of tasks: JAVA - its execution is defined by extending the Proactive JavaExecutable class. NATIVE - its execution can be any user program, a compiled C/C++ application, a shell or batch script. A native task can be specified by a simple command line. MULTI-NODE - execution over more nodes
Job and selection script example
backbone Scheduler service The Job Scheduler GE is the result of an internal collaboration between the Scheduler and Resource Manager (RM) services. The scheduler is in charge of registering jobs submitted and put them in a queue according to a scheduling policy (FIFO by default). In order to launch jobs, the Scheduler Service has to obtain nodes (resources) from the Resources Manager. Once done, it launches an execution of a task on each node. When task is ended, it launches another one and so on and so forth.
Scheduler Portal Potential Look&Feel
backbone RM service Resource Manager is a client/server service Deploys nodes (launch JVMs) automatically to different types of infrastructure (i.e. grid, cloud, another resource manager, etc). Maintains and monitors the list of resources and manages their states (lost, deploying, configuring, free, busy, down). Supplies computing nodes to users based on user criteria (i.e. specific operating system, available resources or licenses and many others).
RM Portal Potential Look&Feel
Futuristic scenario with several protocols
Protocols (in brief) rmi to start, but.. (****=higher, *=lower) AMQP (Advanced Message Queuing Protocol) ready!!..scalable, standard, open-source, interoperable, reliable, load balancing
in Cloud Hosting Chapter Back-end Usage at GEs disposal Cloud Edge is low performance device. That could lead to low computing issue In general each GE, when overloaded, can ask a temporary help, modelled as a job, to Job Scheduler GE If needed, any web application deployed on the cloud proxy can interact with Job Scheduler GE, as well Really useful to accomplish periodic jobs inside a data centre. Basically, Job Scheduler GE is a generic purpose GE, which misses in our Cloud Computing Architecture. Potentially it might act as PaaS or orchestrator. Cloud Computing and current GEs could actually allow an elastic business management in long time scope, but do not cover the needs of a short time or temporary business model.
@FI-ware wiki Main References: Architecture Baseline Assets Installation & Administration Guide User & Programmers Guide It's the time for demos!!...
Our Objective... create a solid basis for the Internet of the Future!