IWFMS: An Internal Workflow Management System/Optmzer for Hadoop Lan Lu, Yao Shen Department of Computer Scence and Engneerng Shangha JaoTong Unversty Shangha, Chna lustrve@gmal.com, yshen@cs.sjtu.edu.cn ABSTRACT The scale of jobs runnng on parallel computaton platform such as Hadoop s ncreasng quckly, thus workflow engnes that manage data processng jobs have become ncreasngly mportant. However, tradtonal workflow management systems are mostly outsders to Hadoop and cannot fulfll many mportant requrements, such as user constrants and schedulng optmzatons. In ths paper, we present IWFMS, an nternal workflow schedulng system/optmzer that 1) Manage the resource allocaton and executon of jobs n workflows to acheve hgher effcency, 2) Schedule workflows to meet a much rcher set of user constrants such as deadlnes, prortes and workflow trgger events. We dscuss the archtecture of ts key components and evaluate ts features and performance. KEYWORDS IWFMS; Hadoop; Workflow Management; Optmzaton; User Constrants. 1 INTRODUCTION Hadoop [1] s a massvely scalable parallel computaton platform capable of runnng many thousands of jobs per day. Wth the user group growng, bg data n ts raw form rarely satsfes the common user and Hadoop developer's data requrements for performng data processng tasks. Frameworks that help automate ths ncreasngly complex process and codfy work nto repeatable unts or workflows that can be reused over tme wthout the need to wrte any new code or steps are essental layer of Hadoop for common user and more complcated seres of work, such as Ooze[2], Azkaban[3]. These systems, known as workflow engnes, provde a mult-tenant servce that effectvely manages dverse jobs wrtten n a varety of tools and languages, and n the same tme, has an nterface whch s user-frendly and qute convenent. These systems also allow users to specfy constrants. For example, model workflow executon trggers n the form of the data, tme or event predcates, the workflow job s started after those predcates are satsfed. However, these workflow engnes are external workflow submttng systems whch submt workflow as ndependent, unrelated jobs and provde very few executon optmzatons. The cluster then schedule these jobs wthout consderng the relatonshp between them, ths desgn structure s a great restrct to the potental performance mprovement of workflow executon, furthermore, the user constrants are also lmted, supported trgger events are usually tme ntervals or data avalablty but cannot be complcated crcumstances that nvolve cluster condtons or schedulng nformaton. These factors have restrcted the scenaros n whch workflow engnes could have been more wdely used. In ths paper, we ntroduce a new management system and optmzer for workflow applcatons, the Internal Workflow Management System (IWFMS [4]). It allows users to buld complex dependency structure lke data transformatons or decson branches out of multple component jobs cost effectvely, whch nhert the superortes of exsted workflow engnes, whle at the same tme, t provdes a rcher set of user ISBN: 978-1-941968-04-8 2014 SDIWC 59
constrant optons, ncreased system utlzaton and a sgnfcant reducton n the workflow completon tme. IWFMS s archtected to work as a plug-n for Hadoop, unlke other exsted workflow mplementatons, IWFMS provdes servce for both users and the clusters. Its core module, the WFScheduler, mantans the cluster nformaton and helps optmze workflow jobs executon, clusters wth IWFMS nstalled can recognze workflow jobs from ordnary jobs and schedule them accordng to ther workflow structures and confguratons. IWFMS can also be used as a sngle optmzer whch wll collaborate smoothly wth man workflow engnes lke Ooze. Organzatons that have been usng tradtonal workflow management systems may not want to change the user nterface for nteractve workflow management and APIs for ntegraton, under these crcumstances, they can abandon the user servce part of IWFMS (although not recommended) and just plug WFScheduler nto Hadoop, ths wll provde most of the workflow executon optmzatons from IWFMS and, wth some manual modfcatons to the confguraton fle, full package of new user constrants. Ths rest of ths paper s organzed as follows: n secton 2 we explore related workflow systems and dentfy ther shortcomngs wth respect to Hadoop requrements. Secton 3 provdes a bref ntroducton to workflow applcatons. Secton 4 descrbes the desgn and mplementaton of IWFMS. In secton 5 we present the expermental setup for measurement and dscuss the results for IWFMS performance. Fnally we summarze our current work and dscuss our future work n secton 6. 2 RELATED WORKS Scentfc workflow on clusters and grds have been studed extensvely, many workflow management systems exst for varous usage needs such as [5-8]. In ths secton, we dscuss a few exstng workflow management systems along wth ther benefts and shortcomngs. Ooze[2] s a Java Web applcaton that combnes multple Hadoop jobs sequentally nto one logcal unt of work. There are two types of Ooze jobs: Ooze Workflow jobs and Ooze Coordnator jobs, the former are a sequence of actons that form a Drected Acyclc Graphs (DAGs), and the latter are recurrent workflow jobs that are trggered by tme and data avalablty. Ooze makes t easy to control over complex jobs and repeat them at predetermned ntervals. However, due to ts complete separaton from clusters, t s more a workflow bulder and auto-submtter than a full-functon workflow management system. It consders lttle about Hadoop clusters condtons and provdes none nformaton about the workflow applcatons to Hadoop, what Ooze does s submttng any jobs as soon as they are ready to be executed, and then leave them to Hadoop wthout executon optmzaton towards ther workflow structures and data dependency. The Pegasus [9] system provdes a comprehensve soluton for constructng and enactng scentfc workflows on the NCSA TeraGrd [10]. Pegasus enables the workflows to be executed locally n a smultaneous manner. It montors the executon performance and provdes mechansms to mplement optmzaton technques. However, Pegasus seems to be not qute sutable for Hadoop clusters, and cannot fulfll the requrements under many scenaros. Azkaban [3] s a batch workflow job scheduler created by LnkedIn to run ther Hadoop jobs. It also provdes an easy to use web user nterface to mantan and track ther workflow applcatons. Azkaban executes all workflow jobs as part of a sngle server process, whch does not support authorzaton for job submsson and control. It shares the same advantages of most exstng management systems that are easy to use and user-frendly, but t provdes even fewer optons for user constrants than Ooze, and lacks sgnfcant features requred for a workflow management system. ISBN: 978-1-941968-04-8 2014 SDIWC 60
There are several other e-scence tools that help construct and execute workflows usng local and/or remote data[11-14], n comparson, IWFMS s dfferent from exstng workflow systems n that part of t works nsde Hadoop, ths consttuton breaks the obstacle between management system and Hadoop, whch leaves far more space for optmzaton technques and user customzatons. 3 BACKGROUND 3.1 Workflow Applcaton A workflow applcaton s a collecton of actons (.e. Hadoop Map/Reduce jobs, Pg jobs, scrpts or executable fles) arranged n a control dependency Drect Acyclc Graph (DAG), specfyng a sequence of actons executon. A launched Hadoop workflow applcaton conssts of a seres of submtted or uncommtted Hadoop jobs due to whether ther dependences are satsfed. A workflow engne usually n charge of construct the dependency DAG and set ts trggers, when the trggers are satsfed, the workflow engne wll start submttng jobs n the workflow applcaton accordng to ther DAG poston. The DAG graph s specfed n some knds of descrptve language, n our mplementaton, we use hpdl, a XML Process Defnton Language, whch s almost the same as what Ooze uses, except that we add some new constrant nodes to the grammar. 3.2 hpdl Language hpdl s a farly compact language, t uses a lmted amount of flow control and acton nodes. Control nodes defne the flow of executon, whch ncludes mechansms to control the workflow executon path lke decson, fork or jon operaton, and begnnng/end of a workflow such as start, end and fal nodes. Acton nodes are the mechansm by whch a workflow trggers the executon of a computaton/processng job, the job can be wrtten n a varety of tools and languages, e.g. Hadoop map-reduce jobs, executable fles, shell or Python scrpts and sub-workflows. 3.3 DAG Path In ths paper, DAG path concept s very mportant to our optmzaton technques. A DAG path of the workflow applcaton contans all the acton nodes (jobs) along a drected path from the start node to the end node. A workflow job may belong to multple paths, and a DAG path s called crtcal path n ths paper f ts jobs get the heavest sum total workload. A workflow applcaton s completon tme s decded by the duraton of crtcal path. 4 FEATURES and IMPLEMENTATION In ths secton, we frst descrbe the overvew of IWFMS's superorty and functons. Then we dscuss how WFScheduler addresses ts key functons n subsequent sub-sectons. 4.1 Desgn Goals Both the workflow jobs and ordnary jobs arrve randomly to the cluster, the former may be drven by WFEngne or external workflow engnes lke Ooze or Azkaban et al. When a job s submtted, by checkng the dstrbuted cache, we frst decde whether t s a workflow job. If the job belongs to a workflow applcaton, we read and analyss the confguraton fle along wth the job submsson fle to get to know whch workflow t belongs to and the poston of the DAG structure t les n, besdes, f the job s the frst job submtted of a workflow applcaton, we also load and mantan the nformaton of the workflow and the user-specfed confguraton, accordng to whch we manage the workflow's lfecycle (such as sleep, nvoke and launch) and optmze ts executon. Durng the workflow executon, we observe the process rate and dynamcally estmate the executon tme of each DAG path to dentfy the crtcal path, then we adjust resource allocaton and data dstrbuton for jobs on each path to acheve a shortest overall executon tme, whch s determned by the crtcal path. As the completon tme tends to be ndcated by montorng the processng cost of jobs on crtcal path, the resource needed for ISBN: 978-1-941968-04-8 2014 SDIWC 61
each jobs of the workflow applcaton can be computed to meet the deadlne (f specfed), one of the desgn goals s to maxmze the number of workflow applcatons whle satsfyng the deadlnes. The desgn goals for IWFMS were: (1) Help Hadoop to recognze jobs submtted that belong to workflow applcatons, be aware of the workflow applcatons' DAG structures and adjust schedulng strategy accordngly. (2) Dynamcally allocate resources n cluster to meet the gven deadlne of workflow applcatons based on the observed progress rate acheved by whose jobs. (3) Dynamcally allocate resources for jobs on dfferent DAG paths of a workflow applcaton to acheve hgher resource utlzaton and shortest total completon tme. (4) Provde more trgger mechansms by mantanng cluster status nformaton and montorng jobs and workflow applcatons. 4.2 IWFMS Archtecture IWFMS conssts of two major parts, WFEngne and WFScheduler. WFEngne provdes user nterface for constructng and submttng workflow applcatons. WFScheduler s the kernel component for IWFMS to mplement workflow executon optmzng mechansms. WFScheduler s developed as a contrb module for Hadoop, and we have mplemented varous versons that support dfferent job schedulng strateges for Hadoop such as Far, Capacty etc. Fgure 1 shows the basc archtecture of the WFScheduler, whch s dvded nto several modules. WFScheduler works as a mddle layer between Hadoop JobTracker and Hadoop scheduler (FIFO, Far, Capacty etc.), t detects jobs of workflow applcatons and manage them whle does not nfluence the schedulng of ordnary jobs. We have separated the orgnal Hadoop schedulng part from WFScheduler and make t a pluggable module, however due to the archtecture of Hadoop tself, the WFScheduler, along wth other scheduler contaned, works together as a pluggable module for Hadoop. Job Added Job Completed Job Updated Lstener Has Cache fles: Workflow.xml Yes hpdl Parser Workflow App Manager Workflow Queues No Comparator Job Queues Hadoop Scheduler New Avaable Jobs Fgure 1 WFScheduler Internal Archtecture The core modules of WFScheduler are: Workflow Manager: Ths module keeps track of submtted workflow applcatons, n charge of managng ther lfecycles and resource allocaton, t also responds to workflow trgger events by addng jobs of the workflow that was trggered to the scheduler ntalze queue. The workflow queue mantans the nformaton about all runnng and watng workflow applcaton, ncludng node dstrbuton and runtme condton. Workflow ProgressRate Watcher: Ths module montor the progress rate of each runnng workflow applcatons, more specfcally, t compute each path's progress rate of the workflow applcatons by observng the runnng cost of jobs on dfferent paths, based on whch t hereby estmate the completon tme of crtcal path and resource needed for other paths. Cluster Watcher: Ths module keeps track of the cluster nformaton. It watches cluster nodes condtons and job runnng stats, gathers mportant ( or regstered ) events then send them to the core center, Workflow Manager, n whch certan reacton wll be determned. hpdl parser: The hpdl parser exsts n both WFEngne and WFScheduler. It serves as a xml format hpdl language fle parser whch extract nformaton needed ISBN: 978-1-941968-04-8 2014 SDIWC 62
about a certan workflow applcaton, such as, nvoke condtons, dependency between jobs, workflow jobs nformaton, user constrants etc. Hadoop Scheduler: The Hadoop scheduler works almost the same as tradtonal Hadoop scheduler, for nstance, Far Scheduler [15] and Capacty Scheduler [16]. The other parts of WFScheduler work as a mddle layer between Hadoop JobTracker and Hadoop scheduler. In fact, most task schedulers for Hadoop, whch serves as pluggable modules, can easly be modfed nto IWFMS s Hadoop scheduler, thus, as mentoned above, IWFMS can support dfferent job schedulng strateges for Hadoop. 4.3 Workflow Job Schedulng The schedulng strategy of workflow jobs and tradtonal jobs dffers n several aspects. There are varous factors that may nfluence the resource allocaton and runnng prorty of a workflow job, for example, the use defned deadlne of workflow applcaton the job belongs to, the total resource needed for the workflow DAG path the job s n, dstrbuton of nput data and the dependency property. We take these factors nto consderaton n order to acheve hgher executon effcency. 4.3.1 Workflow Deadlne Estmaton To estmaton the total duraton of a workflow applcaton we need to assess the completon tme of each workflow paths. The crtcal path, whose jobs have the heavest workload and longest total completon tme, decdes when the workflow applcaton wll fnally complete. In our mplementaton, to balance the progress rate of whole workflow, we need not only the duraton of the crtcal path but also ones of each workflow path wth certan amount of resources allocated. Accordngly, we can dynamcally decde how many resources the currently runnng job on each DAG path need. We develop an ntal estmaton model based a set of assumptons: 1) The cluster conssts of homogeneous nodes, so that the tme cost of processng for each map or reduce task s equal; 2) The tme cost of unntalzed jobs task can be approxmated by the average tme cost of all map or reduce tasks on the DAG path; 3) Input data s ether already avalable n HDFS or wll successfully outputted by dependent job. We extend the estmaton model used by [17-18], and combne technques proposed n [19] to t for workflow schedule envronment. In our workflow estmaton model, we ntroduce Hadoop specfc notatons T, P, J, I, n m, n r, C m, C r, f m, f r, S, as descrbed below: T = {T p1,,t pk }: The estmated complete tme of path. P = (J 1, J 2, J N ): Workflow DAG paths, whch consst of a seres of jobs and connect the start node and the end node. J = (t m1, t m2 t mu,t r1,,t rv ): A Hadoop workflow job that corresponds to an acton node of the workflow DAG. I = { I p1, I p2,,i pk }: I p represents nput data of the frst job of path P. n m, n r : Number of map/reduce slots assgned to job. Cm = {C m J }: Cost of processng a unt data n map task. Cr = {C r J }: Cost of processng a unt data n reduce task. f m : Map flter rato. The fracton of nput that the map process produces as output of job. f r : Reduce flter rato. The fracton of nput that the reduce process produces as output of job. S = { S J}: Start tme of frst map task for path. Our schedulng strategy s based on followng expressons, n whch j means currently the cluster s schedulng the j-th job of path k: ISBN: 978-1-941968-04-8 2014 SDIWC 63
j T Pk = S k + I j 1 J f l m f l l=1 r C m =0 j n m j 1 + I J f l m f l l=1 r C r =0 N n r + I Jj(f m f r ) j C m n m k =j+1 N + I Jj(f m f r ) j C r =j+1 n r k (1) T P T Pj (where P, P j P ) (2) K avaableslots m = nk m k=0 K k avaableslots r = n r k=0 (3) (4) In bref, we calculate how many resources a job on a workflow DAG can get based on the workload ths path left, average process tme and the total amount of resources ths workflow applcaton can get. When a job belongs to multple DAG paths, the slots t can share s decded by the maxmum number of ts paths need dynamcally. 4.3.2 Schedulng Algorthms The optmzaton of workflow jobs s based on followng prncples: 1) Jobs and ther tasks on dfferent paths get only enough resources they need so dfferent paths of a workflow applcaton end at smlar tme, whch makes sure that the path wth the heavest work load get the most resources and hghest runnng prorty comparng to jobs on other paths of ths workflow applcaton. 2) Based on workflow DAG structure and jobs data dependency, tasks are scheduled to be data-local to the utmost. 3) Workflow applcatons that are n sleep state watng for user specfed events should release ther resources and take only mnmum space, by whch approach the workload of scheduler wll be kept n an acceptable level. To acheve these goals, the WFScheduler sometmes need to temporarly hold certan resources and reserve them for the rght job to be launched or ntalzed. Shown n Algorthm 1, we mplement a custom comparator to mantan the prorty of slots allocaton between jobs of same or dfferent workflow applcatons. Durng the task assgnment, we dynamcally calculate the slots need for the next job to execute on the path and start to reserve slots before the current job ends, through ths method, IWFMS ensures a sgnfcantly hgher percentage of local-tasks. Compared to the tme saved of communcaton cost of ntermedate data, the temporary ISBN: 978-1-941968-04-8 2014 SDIWC 64
reservaton cost can almost be gnored, whch we wll testfy by experments n Secton 5. The task assgnment and reservaton strategy s presented n Algorthm 2, the pseudo-code descrbes the man dea of the technques, whle n practce, ths strategy s actually mplemented by multple modules and s hard to be covered n one sngle functon. 5 EXPERIMENTAL RESULTS AND ANALYSIS In ths secton, we evaluate the benefts of IWFMS, whch contan the performance mprovement of workflow applcatons and man IWFMS characterstcs, ncludng user customzatons lke deadlne and trggers. We ran seres of Hadoop workflow jobs that had unbalanced DAG structure and data dependency between jobs, whch are common type of operaton performed by MapReduce. Then we added varous types of constrants lke deadlne, prortes and trgger events to the submtted workflow applcaton. 5.1 Expermental Setup Snce buldng an ndustral grade dstrbuted system wth thousands of machnes s beyond the scope of our ablty, to evaluate the performance of IWFMS, our experments were conducted n an expermental cluster whch conssted of 4 physcal nodes wth 3 as TaskTrackers and 1 as JobTracker. Each node specfcaton was: 63GB man memory, 23 Intel Xeon 2.40GHz processors runnng Ubuntu. Hadoop verson 1.2.1 was used n the cluster, along wth Ooze verson 3.2.0 on another node as baselne system. TaskTrackers had 8 map slots and 4 reduce slots, most confguraton parameters were default values. As baselne system, we nstalled Ooze verson 3.2.0 on another machne whch was used for submttng workflow jobs. In the baselne system, workflow jobs ran on the same cluster whle wthout IWFMS nstalled. 5.2 Results 5.2.1 Performance Improvement Fgure 2 and Fgure 3 shows the task allocaton results for the same workflow applcaton runnng n cluster wth or wthout IWMFS. We can see that the tasks allocaton s done qute dfferently, n IWFMS, jobs on lght-workload path get lmted amount of resources whle ones on crtcal path can obtan more slots, the slots lmts make each path of the workflow ISBN: 978-1-941968-04-8 2014 SDIWC 65
applcaton ends at smlar tme. On the other sde, we notce that jobs runnng n tradtonal Hadoop system wthout IWFMS acqure slots based only on the tme they are launched, whch leads to a lower utlzaton rate and longer total tme cost. Table 1 Consttuents of workflow applcatons Exp1 Exp2 Exp3 Exp4 Exp5 Exp6 Balanced 100% 80% 60% 40% 20% 0% Unbalanced 0% 20% 40% 60% 80% 100% Fgure 2 Task allocaton n Hadoop+Ooze Fgure 4 Performance of IWFMS To evaluate the performance mprovement further, we ran a seres of workflow applcatons from Ooze examples that were slghtly modfed and had dfferent DAG structures. Fgure 4 llustrates that compared to baselne system; IWFMS exhbts 12%-36% makespan reducton. 5.2.2 Meetng workflow deadlnes Fgure 3 Task allocaton n Hadoop+IWFMS Another mportant reason for the performance speedup s dstrbuton of slots allocated. In Hadoop wth IWFMS, the majorty of workflow tasks are data-local tasks, thus there are much fewer data transfer cost due to more local computatons, whch would beneft executon effcency for both balanced and unbalanced workflow applcatons. Fgure 5 Task allocaton under dfferent deadlnes In our next experment we tested the effectveness of workflow deadlne constrant. Fgure X shows the task allocaton results for the same workflow applcaton run-nng on ISBN: 978-1-941968-04-8 2014 SDIWC 66
cluster wth dfferent deadlnes or wthout deadlnes specfed. For 400s deadlne, the task lmt s 20; whle for 800s deadlne, maxmum tasks allocated are lmted to be no more than 10-12. In both cases wth deadlnes specfed, the workflow deadlne s met and the WFScheduler ensures that durng the workflow applcatons executon, enough slots are met for each path of the workflow. 5.2.3 Other user constrants 6 CONCLUSION and FUTURE WORKS In ths paper, we ntroduce our mplementaton of workflow management system that has brand new desgn archtecture. Unlke other workflow engnes, t cares about when and where the workflow jobs wll execute and knows about the condtons of Hadoop cluster. IWFMS provdes us more effcent executon of workflow jobs, at the same tme, workflow applcatons are no longer offlne jobs wth the applyng of IWFMS. Future work ncludes desgnng and mplementng a scrpt language for the user constrant operatons, through whch users wll gan much greater control to ther workflow applcaton. REFERENCES Fgure 6 Task allocaton wth trgger event set Fgure 6 shows the workflow executon when we set trgger event(s), n ths case, the trgger event s when some jobs whose names contan one partcular strng are done, we can see that before the trgger event happened, the workflow applcaton was asleep, and then the trgger event nvoked t and WFScheduler added t to the runnng queue. In fact, the trgger event confguraton s qute flexble and can be easly set n the confgure fle submtted along wth job fles, for example, we can assgn some workflow applcatons to be executed only when cluster s n lghtworkload, what we need to do s set the global confguraton secton n workflow.xml fle. However these user constrants cannot be satsfed usng other workflow engnes lke Ooze and Azkaban, because they don t mantan any nformaton about the cluster and Hadoop, n fact, they don t even stop submttng jobs when Hadoop s not n servce. [1] Apache Hadoop. (n.d.). Retreved October 27, 2014 from http://hadoop.apache.org/ [2] Apache Ooze. (n.d.). Retreved October 27, 2014 from http://ncubator.apache.org/ooze/ [3] The Azkaban Project. (n.d.). Retreved October 27, 2012 from http://gthub.com/azkaban/azkaban [4] The IWFMS Project. (n.d.). Retrved October 27, 2014 from https://gthub.com/lustrve/workflowscheduler [5] Bowers, S., & Ludäscher, B. (2005). Actor-orented desgn of scentfc workflows. In Conceptual Modelng ER 2005 (pp. 369-384). Sprnger Berln Hedelberg. [6] Ludäscher, B., Altntas, I., Berkley, C., Hggns, D., Jaeger, E., Jones, M.,... & Zhao, Y. (2006). Scentfc workflow management and the Kepler system. Concurrency and Computaton: Practce and Experence, 18(10), 1039-1065. [7] Majtha, S., Shelds, M., Taylor, I., & Wang, I. (2004, July). Trana: A graphcal web servce composton and executon toolkt. In Web Servces, 2004. Proceedngs. IEEE Internatonal Conference on (pp. 514-521). IEEE. [8] Onn, T., Greenwood, M., Adds, M., Alpdemr, M. N., Ferrs, J., Glover, K.,... & Wroe, C. (2006). Taverna: lessons n creatng a workflow envronment for the lfe scences. Concurrency and Computaton: Practce and Experence, 18(10), 1067-1100. [9] Sngh, G., Su, M. H., Vah, K., Deelman, E., Berrman, B., Good, J.,... & Mehta, G. (2008, January). Workflow task clusterng for best effort systems wth Pegasus. In Proceedngs of the 15th ACM Mard Gras conference: From lghtweght mash-ups to lambda grds: Understandng the spectrum of dstrbuted computng requrements, applcatons, tools, nfrastructures, nteroperablty, and the ncremental adopton of key capabltes (p. 9). ACM. [10] Catlett, C. The phlosophy of TeraGrd: buldng an open, extensble, dstrbuted TeraScale faclty. ISBN: 978-1-941968-04-8 2014 SDIWC 67
Cluster Computng and the Grd 2nd IEEE/ACM Internatonal Symposum CCGRID2002, 2002. [11] Deelman, E., Gannon, D., Shelds, M., & Taylor, I. (2009). Workflows and e-scence: An overvew of workflow system features and capabltes. Future Generaton Computer Systems, 25(5), 528-540. [12] Kepler. (n.d.). Retreved October 27, 2014 from https://kepler-project.org/ [13] Wang, J., Crawl, D., & Altntas, I. (2009, November). Kepler+ Hadoop: a general archtecture facltatng data-ntensve applcatons n scentfc workflow systems. In Proceedngs of the 4th Workshop on Workflows n Support of Large-Scale Scence (p. 12). ACM. [14] Taverna (n.d.). Retreved February 10, 2012 from http://edutechwk.unge.ch/en/taverna_workbench [15] Zahara, M. (2010). The hadoop far scheduler. [16] Hadoop s Capacty Scheduler http://hadoop.apache.org/core/docs/current/capacty_ scheduler.html [17] Kc, K., & Anyanwu, K. (2010, November). Schedulng hadoop jobs to meet deadlnes. In Cloud Computng Technology and Scence (CloudCom), 2010 IEEE Second Internatonal Conference on (pp. 388-392). IEEE. [18] Ln, X., Lu, Y., Deogun, J., & Goddard, S. (2007, Aprl). Real-tme dvsble load schedulng for cluster computng. In Real Tme and Embedded Technology and Applcatons Symposum, 2007. RTAS'07. 13th IEEE (pp. 303-314). IEEE. [19] Cao, J., Jarvs, S. A., San, S., & Nudd, G. R. (2003, May). Grdflow: Workflow management for grd computng. In Cluster Computng and the Grd, 2003. Proceedngs. CCGrd 2003. 3rd IEEE/ACM Internatonal Symposum on (pp. 198-205). IEEE. ISBN: 978-1-941968-04-8 2014 SDIWC 68