The Packing Server for Real-Time Scheduling of MapReduce Workflows

Transcription

1 The Packng Server for Real-Te Schedulng of MapReduce Workflows Shen L, Shaohan Hu, Tarek Abdelzaher Unversty of Illnos at Urbana-Chapagn {shenl3, shu7, Abstract Ths paper develops new schedulablty bounds for a splfed MapReduce workflow odel. MapReduce s a dstrbuted coputng paradg, deployed n ndustry for over a decade. Dfferent fro conventonal ultprocessor platfors, MapReduce deployents usually span thousands of achnes, and a MapReduce ob ay contan as any as tens of thousands of parallel segents. State-of-the-art MapReduce workflow schedulers operate n a best-effort fashon, but the need for real-te operaton has grown wth the eergence of real-te analytc applcatons. MapReduce workflow detals can be captured by the generalzed parallel task odel fro recent real-te lterature. Under ths odel, the best-known result guarantees schedulablty f the task set utlzaton stays below 50% of total capacty, and the deadlne to crtcal path length rato, whch we call the stretch, surpasses 2. Ths paper proves ths bound further by ntroducng a herarchcal schedulng schee based on the novel noton of a Packng Server, nspred by servers for aperodc tasks. The Packng Server conssts of ultple perodcally replenshed budgets that can execute n parallel and that appear as ndependent tasks to the underlyng scheduler. Hence, the orgnal proble of schedulng MapReduce workflows reduces to that of schedulng ndependent tasks. We prove that the utlzaton bound for schedulablty of MapReduce workflows s U B, where U B s the utlzaton bound of the underlyng ndependent task schedulng polcy, and s a tunable paraeter that controls the axu ndvdual budget utlzaton. By leveragng past schedulablty results for ndependent tasks on ultprocessors, we prove schedulable utlzaton of DAG workflows above 50% of total capacty, when the nuber of processors s large and the largest server budget s (suffcently) saller than ts deadlne. Ths surpasses the best known bounds for the generalzed parallel task odel. Our evaluaton usng a Yahoo! MapReduce trace as well as a physcal cluster of 46 achnes confrs the valdty of the new utlzaton bound for MapReduce workflows. I. INTRODUCTION The past decade has seen MapReduce [ 5] becoe the donant dstrbuted coputng paradg n ndustry. The portance of eetng deadlnes of MapReduce workflows has grown n recent years as well, drven by the advent of real-te analytcs [6 6]. The success of the MapReduce county n addressng real-te constrants, however, reans lted due to the nherent dffculty of the workflow schedulng proble on parallel resources. On the other hand, n real-te schedulng lterature, recent results on schedulablty of generalzed parallel tasks do not offer hgh platfor utlzaton. The prospect of provng schedulablty bounds of generalzed parallel tasks on ultprocessors n the subcases relevant to MapReduce workflows otvates the work reported n ths paper. The MapReduce dstrbuted coputng paradg splts source data nto ndependent chunks and processes the usng two phases: the ap phase apples the ap functon onto each chunk, generatng nteredate key-value pars, whle the reduce phase aggregates and suarzes those key-value pars based on ther keys. The reduce phase cannot start untl the ap phase fnshes, and both phases can be parallelzed to run on a large nuber of slots, where a slot s a resource unt n MapReduce clusters. MapReduce deployents usually span thousands of achnes, connected by a hgh-speed and hgh-bandwdth ntranet. Assung a bounded network delay (that we can subtract fro the end-to-end deadlne), the platfor acts as a very large ultprocessor. One MapReduce ob ay contan tens of thousands of parallel segents [7, 8]. Due to nput/output dependences that are often requred to carry out coplex algorths, MapReduce obs usually for Drected Acyclc Graphs (DAG), called MapReduce workflows. Many data processng algorths used n the context of MapReduce workflows are bulk algorths (as opposed to ncreental-update algorths). They requre a bulk of data to be present at once. Ths leads to a perodc nvocaton odel, where a volue of data s frst collected wthn the current te-slce, and then the MapReduce workflow s nvoked. Beng able to eet workflow deadlnes s often of crucal portance to busnesses, because applcatons supported by producton MapReduce workflows, such as advertseent placeent optzatons, user graph storage parttons, and personalzed content recoendatons, usually drectly affect ste perforance and copany revenue. The state-of-the-art MapReduce workflow schedulers, such as Ooze [9] and WOHA [8], operate n a best-effort fashon, offerng lttle guarantees on workflow copleton tes. Wth the surge of nterest n real-te workflow executon, recent work addressed schedulng extensons that offer resource parttonng [20, 2], reduce preepton cost to support prortzaton [22], or take deadlnes as nput to the scheduler [, 2]. However, these attepts fall short of offerng tng guarantees. Adnstrators cannot tell, quanttatvely, what axu utlzaton ther MapReduce clusters can bear before obs start laggng too far behnd. Ths calls for an analytcally well-otvated schedulng and adsson control polcy, whch s the topc of ths paper. Ths paper akes two contrbutons. Fro the perspectve of MapReduce applcatons, we offer an analytc result and a run-te echans to guarantee schedulablty of MapReduce workflows as long as a schedulablty bound s et.

2 Fro the perspectve of real-te foundatons, we prove the best known bound for schedulablty of the generalzed parallel task odel n a specal subcase of relevance to MapReduce applcatons. The contrbutons rest on the novel dea of the Packng Server. It consttutes a run-te echans that akes concurrent precedence-constraned workflows look lke ndependent perodc tasks to the underlyng MapReduce scheduler. We then derve a converson factor that expresses a bound for schedulablty of MapReduce workflows as a functon of the underlyng utlzaton bound for schedulablty of ndependent perodc tasks. Detals of a MapReduce workflow can be captured by a generalzed parallel task odel [23]. Aong the etrcs used n exstng work, utlzaton bounds and capacty augentaton bounds gve rse to effcent adsson control polces, as they only requre sple nforaton about the task set and the platfor. The best-known result for schedulablty of workflow tasks, that s aenable to a sple adsson control polcy, s a capacty augentaton bound of 2 for plct-deadlne task odels usng a federated schedulng strategy [24]. In order to guarantee schedulablty, ths bound requres the task set utlzaton to be less than 50% of total capacty, and the deadlne to crtcal path length rato, whch we call the stretch (), to be larger than 2. Ths constrant (only 50% of capacty) ay be too restrctve for soe systes. Fortunately, ore advances have been ade on schedulablty of ndependent task sets, achevng uch hgher utlzaton bounds, especally when the nuber of processors s large and the ndvdual unts of work are sall. For exaple, López et al. [25] proved that frst-ft parttoned EDF (EDF-FF) can schedule any syste of ndependent perodc tasks on processors, gven that the total capacty stays below U B + +, where s nverse of the axu ndvdual task utlzaton (.e., u ax ). As another exaple, the global EDF schedulng [26] guarantees to eet all deadlnes ( f ) the total task set utlzaton s less than U B +. Note that, for u ax, the total schedulable utlzaton s larger than /2 (.e., larger than 50% of total capacty). In fact, the bound approaches 00% of capacty as ncreases. These observatons suggest that task set abstractons or transforatons that ake workflows look lke ndependent tasks ay prove schedulablty. The Packng Server echans, presented n ths paper, s nspred by the above observaton. Each Packng server conssts of a nuber of budgets dedcated to a gven MapReduce workflow. The MapReduce syste schedules these budgets as ndependent tasks. When nvoked, the budget runs MapReduce segents n a anner that respects workflow precedence constrants. Hence, the orgnal proble of analyzng schedulablty of MapReduce workflows on ultprocessors s translated nto the well-known proble of analyzng schedulablty of ndependent tasks. The utlzaton bounds for the latter are well known both for EDF and fxed prorty schedulng, as well as both for parttoned and global schedulers. We prove a converson factor between the utlzaton bound for schedulablty of MapReduce workflows acheved by our schee, and the utlzaton bound of the underlyng scheduler for ndependent tasks. Naely, the MapReduce workflow s schedulable f utlzaton s below U B. In the followng, we shall use the conventon of expressng utlzaton as percentage of total cluster capacty. Hence, for exaple, for a cluster of achnes, we shall say 50% when we ean /2, and wll refer to t as the cluster utlzaton. Snce the deadlnes for MapReduce workflows are typcally large (e.g., hours) and the clusters are bg, t s coon that MapReduce workflows enoy a large stretch,, leadng to a hgh utlzaton bound. Hence, we prove the best known results for schedulng MapReduce DAGs on ultprocessors. The paper descrbes how to sze Packng servers, derves the schedulablty bounds attaned, and presents the polces used nsde a server n handlng MapReduce workflows. Evaluaton results confr the proved schedulablty. The reander of ths paper s organzed as follows. Secton II brefly ntroduces the MapReduce ob odel. Secton III and IV develop the converson factor for ndvdual MapReduce obs and workflows, respectvely. We descrbe the applcaton-level schedulng algorth for packng work nsde servers n Secton V. Secton VI presents evaluaton results. We survey the related work n Secton VII. Secton VIII concludes the paper. II. MAP-REDUCE WORKFLOW MODEL We refer by workflow task sets to those task sets that contan nter-dependent sequental obs. Ths s n contrast to ndependent task sets n whch no ob dependences are present. The hgh-level dea of our technque s to transfor a MapReduce workflow task set τ nto an ndependent task set τ, pleented as Packng server budgets, such that the schedulablty of τ s a suffcent condton for the schedulablty of τ. The transforaton s done at the cost of ntroducng ncreased (vrtual) coputaton tes n τ, leavng τ at a hgher utlzaton than τ. We show that the utlzaton of τ s at ost tes larger than τ, where s a tunable paraeter that controls the axu ndvdual server budget sze. Hence, f an ndependent task set scheduler A offers a utlzaton bound U B, the MapReduce workflow set τ can eet all deadlnes provded that ts utlzaton stays below U B. The MapReduce lterature uses ternology dfferently fro the real-te lterature, leadng to a potental confuson over what s eant by such ters as obs and tasks. In ths paper, we follow the defntons coon n real-te lterature as uch as possble. We say that a MapReduce ob conssts of a ap phase and/or a reduce phase. It ay be

3 that a MapReduce ob contans only a sngle ap or reduce phase, although coonly t contans both. Each phase contans ultple segents that ay execute n parallel. We call a segent a apper or a reducer dependng on whether t belongs to a ap phase or a reduce phase. The executon of a apper or a reducer occupes a resource slot (or ust slot), whch could, for exaple, be a core n a ult-core platfor. Wthn one ob, no reducer ay start before all appers fnsh. A MapReduce ppelne chans ultple MapReduce obs together, resultng n a sequence of phases. In a general MapReduce workflow, MapReduce obs collectvely for a Drected Acyclc Graph (DAG), where each node represents a MapReduce phase and each edge ponts fro node a to node b represents the dependency that phase b cannot start before phase a fnshes. A MapReduce workflow task τ s a perodc task that generates a MapReduce workflow every T te unts wth relatve deadlne D. We denote the nuber of segents (appers or reducers) at the th phase of the workflow of task τ by, and the worst-case coputaton te of an ndvdual segent by c. A MapReduce workflow task set τ contans ultple MapReduce workflow tasks. Usually, the nput of a MapReduce workflow task nvocaton depends on the output of the prevous workflow nvocaton fro the sae task, resultng n an plct-deadlne task odel (D T ). We defne the stretch of workflow task τ, denoted by, as the rato of relatve deadlne D over crtcal path length, denoted by L. Let denote the nu stretch of all workflow tasks, n{ }. Note that, f the workflow contans a sngle path, path (.e., t s a ppelne), L path c sued over the path. The crtcal path n a DAG workflow s the longest executon path n the DAG. Hence, L ax pathk DAG path k c Please note that the workflow odel enoys the sae expressveness as the generalzed parallel task odel [23], as any nstance of the latter odel can be transfored nto a workflow odel by constructng a sngle-apper MapReduce ob for each node n the generalzed parallel task odel. Hence, the above ternology s ntroduced erely for seantc convenence of appng results to the MapReduce applcaton world. Dfferent fro the typcal ultprocessor scenaro, the MapReduce platfor usually spans thousands of achnes [27], and a sngle phase ay contan as any as 30 thousand segents [7, 8]. Ths encourages us to pay specal attenton to cases where s large. Moreover, snce deadlnes and parallels are large, we are nterested n scenaros of large stretch,. III. THE PACKING SERVER UTILIZATION BOUND In ths secton, we restrct each workflow to contan a sngle MapReduce ob. The analyss s generalzed to ppelnes and DAGs n Secton IV. budgets Packng Server Packng Server 2 Packng Server n Schedulng Sgnals Schedulng Sgnals... 2 n budgets... 2 n... budgets Underlyng Independent Task Scheduler (e.g., EDF-FF) Fgure : Packng Server Archtecture A. The Packng Server For each task τ n the MapReduce task set τ, we propose to create a Packng server, τ, gven by several parallel budgets, where the sze of each budget of server τ s denoted c and the nuber of budgets (also called, server concurrency). The nae, Packng server, was chosen because the server packs segents of the orgnal workflow task nto a saller set of budgets. The set of Packng servers s collectvely called set τ. As we show later n ths paper, a property of how segents are packed nto budgets s that these budgets can be scheduled by the underlyng scheduler as f they were ndependent tasks. They can be grated aong cores, preepted, and prortzed as the underlyng schedulng polcy requres, wthout pactng the ablty of the Packng server to respect workflow synchronzaton (.e., segent precedence) constrants. Clearly, the budgets have to be szed such that: () collectvely, they ft all workflow segents and () ndvdually, they ft the crtcal path of the workflow. Below, we descrbe how budget sze s chosen, then prove a converson factor bound that expresses the utlzaton bound for schedulablty of workflows n ters of the utlzaton bound of the underlyng scheduler. Fgure shows the hgh-level archtecture of how Packng servers work. Note that, all budgets n the sae Packng server have the sae budget sze. A Packng server s vald f ts budget sze s saller than the relatve deadlne of the orgnal MapReduce ob (c D ). B. The Case of a Sngle Job We frst consder the specal case where the workflow task, τ, s coposed of a sngle path represented by a successon of one ap phase and one reduce phase. We gve a saller ndex to the phase wth the larger nuber of segents. Hence, 2. Wthout loss of generalty, we assue that s a ap phase. (The dscusson apples equally f was a reduce phase.) In choosng server concurrency,, we note that soe ndependent task schedulng algorths on ultprocessors Schedulng Sgnals

4 Map Segent Reduce Segent Budgets Fro Vrtual Segents Budgets Fro Map Phase Budgets Fro Reduce Phase t t t t (a) MR Job a (b) Packng Server for Job a (c) MR Job b (d) Packng Server for Job b Fgure 2: Iproved Packng Server Constructon Schee are senstve to the axu ndvdual task utlzaton u ax [25, 26]. The saller the axu ndvdual task utlzaton, the better the schedulablty bound. Hence, we ntroduce a tunable paraeter,, to curb u ax of converted ndependent server budgets. Intutvely, the Packng server treats D D as the worst-case allowable budget sze, guaranteeng ndvdual budget utlzaton to be upper bounded by (.e., u ax D D ). In order to pack a MapReduce ob nto an nterval of D te unts, we derve two lowerbounds on Packng server concurrency,, that ste fro the followng condtons: The total WCET condton: In order to ft all orgnal coputaton of the workflow task nto budgets of length no longer than D, we should satsfy: c + 2 c2 () The crtcal path condton: In order to allow the MapReduce ob to fnsh n D te unts, the phase wth ore segents (phase accordng to our ndexng) has to fnsh n D c2 te unts. Therefore: c D (2) c2 Please note, that these two lower bounds do not donate each other. For exaple, n Fgure 2 (a), the MapReduce ob contans 7appers of WCET c 3, and 2 5 reducers of WCET c 2 2. Deadlne D was 0, and s set to(.e., D D ). In ths exaple, the frst lower bound wns, as t results n c +2 c2 D 4, whereas the second lower bound leads to c 3. D c2 Fgure 2 (c) depcts another exaple, where the orgnal MapReduce ob conssts of 6 appers of WCET c 3, and 2 reducers of WCET c 2 2. Under ths confguraton, the frst lower bound results n 2 budgets, whereas the second s 3budgets. Hence, we have the followng two cases: Case : When c < D c +2 c2 c2 D, the MapReduce ob concentrates to budgets of budget sze c +2 c2 D D C C /D, where C c + 2 c2. Fgures 2 (a)-(b) depct an exaple. As ths constructon strategy ntroduces no extra coputaton, the result Packng server τ shares the sae utlzaton wth ts orgnal MapReduce task τ (u u ). Case 2: When c D c +2 c2 c2 D, the reduce phase orgnally has less segents than c. Therefore, we add c D c2 2 D c2 vrtual reducers, as shown n Fgures 2 (c)-(d). Together wth vrtual reducers, the orgnal MapReduce ob converts to c budgets. In each budget, the ap phase and the reduce phase D c2 (wth vrtual reducers) contrbute c c /(D c2 ) and c2 executon te respectvely, resultng n the budget sze of c c c /(D c2 ) + c2. C. Converson Penalty It s crucal that we bound the utlzaton penalty ntroduced durng Packng server constructon, whch drectly affects the converson factor when brdgng schedulablty utlzaton bound fro ndependent tasks to MapReduce tasks. Lea. The utlzaton (u ) of the Packng server τ s at ost u, and the axu ndvdual ndependent task utlzaton s at ost, where s the stretch of MapReduce task τ, and [,] s a tunable paraeter. Proof. We prove the lea holds for the two constructon cases separately: Case : c < D c +2 c2 c2 D. As the Packng server constructon procedure ntroduces no utlzaton penalty n ths case, the utlzaton of the Packng server τ equals the utlzaton of ts orgnal MapReduce task τ (.e., u u < u ). Therefore, the lea holds for case. Case 2: c D c +2 c2 c2 D. The concurrency s. Hence, the nuber of vrtual c D c2

5 ( ) reduce segents s c 2 D c2, each of length c 2. Defne η to be c. Then, we have: c 2 u u u ( ) c 2 D c2 c 2 c + 2 c2 ( c D c2 ( c D c2 c ) + c2 c 2 c + c2 η ) ( η D +)( c 2 η ( ( η +) D c 2 η ( η +)( η c 2 ) as 2 ) as D D + ) as η η + η +( ) Reorganzng the result fro Inequalty 3, we have: D c + c2 u u. (4) Hence, Lea holds for both Packng server constructon cases. Lea drectly leads to a converson factor bound of, where n{ }. IV. MAPREDUCE WORKFLOW BOUNDS Real world MapReduce applcatons usually call for ultple MapReduce obs that for a ppelne or a DAG to accoplsh coplex ssons. In the ppelne odel, MapReduce obs are chaned together one after another, resultng n a sequence of phases. The DAG odel s ore generalzed such that the dependences aong MapReduce obs ay for a drected acyclc graph. In ths secton, we frst dscuss how to generalze the utlzaton penalty bound n Lea to MapReduce Ppelnes. Then we show that any MapReduce DAGs can be transfored nto a ppelne wth the sae crtcal path length L and utlzaton u, plyng that the utlzaton penalty bound for MapReduce ppelnes also apples to MapReduce DAGs. A. MapReduce Ppelnes MapReduce ppelne connects n MapReduce obs one after another, resultng n no ore than 2n ap/reduce phases. Gven a nuber x, f >x, phase s called (3) a x-large phase. Otherwse, t s a x-sall phase. Usng the slar strategy descrbed n Secton III-A, the Packng server concentrates the total WCET ( of each ) x-large phase nto x dentcal segents, and adds x vrtual segents to each x-sall phase. After that, the Packng server concatenates each (vrtual) segent wth another (vrtual) segent n the next phase, resultng n x budgets. Bnary search can be used to fnd the nu x such that the budget sze c does not exceed D D. That s to say, usng only x budgets would volate the deadlne D. Then, we have: { } c + c D. (5) { < } The defntons of the deadlne D and the stretch further lead to the followng nequalty: D D c. (6) Cobnng Inequaltes (5) and (6), we have: { } Subtractng { } c + c { < } { < } c c c. (7) fro both sdes, we obtan: c { < } c c c ( ) c (8) Based on Inequalty (8), the total aount of coputaton requreent can be bounded fro below: c { } c ( ) ( ) c (9) Accordng to Inequalty (8) and (9), we have:

6 u u u { <} ( ) ( ) ( ) c c { <x} c c { <} c ( ) ( ) Therefore, the sae utlzaton penalty bound for MapReduce ppelnes. B. Transforng DAGs nto Ppelnes c (as ) (Equaton 9) (0) holds Ths secton further generalzes the sae utlzaton penalty bound to MapReduce DAGs by transforng a MapReduce DAG nto a MapReduce ppelne. There are any dfferent ways to transfor a MapReduce DAG nto a MapReduce Ppelne. One nave soluton would perfor a topology sort on the DAG, and execute phases one after another accordng to the sorted order. However, ths soluton enlarges the crtcal path length L, leadng to a saller after converson, and hence a larger utlzaton penalty bound. Our goal s to develop a strategy that leads to the lowest utlzaton penalty bound. As shown above, the utlzaton penalty bound for a MapReduce ppelne s, whch decreases wth the ncrease of D L. Ths nspres us to desgn an algorth that nzes the ppelne crtcal path length L durng Packng server constructons. The resultng ppelne length s bounded fro below by the crtcal path length L of ts orgnal MapReduce DAG, where the utlzaton penalty ntroduced by Packng servers s nzed. Ths can be acheved by allowng each node n the DAG to start executon as soon as all ts prerequste nodes fnsh. To keep the result as a vald ppelne, a synchronzaton pont s nserted when each DAG node fnshes. Fg. 3 shows an exaple, where the orgnal DAG ob s depcted n (a) and the resultng ppelne s shown n (b). As phases are of dfferent lengths, a node n the orgnal DAG ay break nto ultple phases n the ppelne. For exaple, node 2 and node 5 both follow phase. Hence, these two nodes ay start at the sae te. However, as node 5 fnshes sooner than node 2, ts endng synchronzaton pont breaks workflow of node 2 nto two ppelne phases. The pseudo code s descrbed n Algorth. Algorth Transfor MapReduce Workflows to MapReduce Ppelnes Input: MapReduce DAG task set τ Output: MapReduce ppelne task set τ : procedure TRANS-D(τ) 2: τ 3: for τ τ do 4: Sync {0} 5: for node n τ do 6: l the earlest possble te that the node could start 7: Lay out all segents n node n te nterval [l, l + c ] 8: Sync Sync {l + c } 9: end for 0: sort Sync followng ncreasng order : τ 2: for 2 Sync do 3: P create a phase encapsulates all segents (portons) fall n te nterval [Sync[-], Sync[]) 4: τ τ {P } 5: end for 6: τ τ τ 7: end for 8: schedule τ usng the MapReduce ppelne schedulng algorth. 9: end procedure Algorth loops over all workflow tasks on lnes 2-7. For each task τ, the algorth frst coputes ts synchronzaton ponts on lnes 4-0. As shown on lnes 6-8, each node n task τ assocates wth a synchronzaton pont l + c, where l s ts longest precedng WCET path. Lnes 2-5 dvde the workflow nto a ppelne of phases usng synchronzaton te ponts n set Sync. The result can be vewed as a sngle ppelne task set, τ. Note that, Algorth transfors a workflow task set nto a ppelne task set wthout ncreasng ts utlzaton. Therefore, the sae utlzaton bound U B apples to MapReduce workflows, where n{ τ τ}. V. SCHEDULING MAPREDUCE WORKFLOWS Prevous sectons ntroduce the strategy to convert a set of MapReduce workflow tasks nto budgets that belong to a set of Packng servers, one per task. Those budgets can then be scheduled as ndependent sequental obs by the underlyng scheduler. Each budget s used to execute workflow segents. We now descrbe how the executon order of segents s deterned. Before proceedng wth our descrpton, t s good to understand the dfferences between tradtonal operatng syste schedulng and MapReduce schedulng, whch n our syste s based on Hadoop [2, 3]; an open-source MapReduce pleentaton. These dfferences are portant to the understandng of our workflow scheduler pleentaton. In an operatng systes context, whch has been the tradtonal schedulng context n real-te systes lterature, servers typcally refer to applcaton tasks. The underlyng scheduler typcally refers to an OS kernel scheduler [28]. Ths OS kernel scheduler has the power to allocate physcal resources to tasks. When t nvokes a server, a secondlevel scheduler (pleented n user space) nsde the server

7 2 3 c c c c c 5 5 (a) 6 2 c t Segents Synchronzaton Correspondng Pont DAG node (b) Fgure 3: Transfor a DAG nto a ppelne: (a) shows the orgnal DAG, and (b) shows the resultng ppelne. The ppelne conssts of 6 phases contanng 2, 9, 3, 4, 3, 2 segents and of lengths 5, 5, 2, 3, 3, 2 respectvely. decdes on the order n whch server budget s allocated to dfferent coputatons. In a MapReduce context, the pcture s slghtly dfferent. Frst, all schedulng s done n user space, snce Hadoop s an applcaton-level pleentaton. Hence, the underlyng scheduler refers to a user-level resource anager that perfors coarse-graned resource assgnents. In Hadoop v2 (YARN) [3], the resource anager assgns resources to the so called applcaton asters. In ths case, an applcaton aster acts as a Packng server. The syste can be confgured to have a sngle aster per workflow task. The applcaton aster pleents the second-level scheduler that decdes on the order of executon of MapReduce segents of the correspondng workflow task wthn the server s budgets. In an portant departure fro OS schedulng, n Hadoop, the applcaton asters are assued to be cooperatve. Hence, the schedulng polcy used by the Hadoop resource anager (.e., the underlyng scheduler) s expressed to the applcaton asters as an exact telne showng when the correspondng workflow task s allowed to run and on whch resources. Applcaton asters are therefore clarvoyant about ther exact future schedules. It s ths clarvoyance that allows us to pleent the abstracton of ndependent budgets, such that the underlyng scheduler (the resource anager) does not need to know anythng about workflow topologes and precedence constrants. More specfcally, once the resource anager nfors applcaton asters of ther budget schedules, snce each applcaton aster (.e., Packng server) of a workflow task knows when ts budgets are scheduled, t can deterne a sequence of synchronzaton te ponts wthn ts budgets, Sync {t 0,t,..., t n } such that t 0 0, and the total sze of scheduled budgets fallng n [t,t ) equals the total WCET of the th phase ncludng potental vrtual segents (c ax{, }). The te nstance t thus becoes the te when phase should end and phase begn. The applcaton aster (.e., Packng server) then packs segents of phase nto budget portons fallng n [t,t ). The result of such packng s shown n Fgure 4 for an exaple server coposed of four budgets. Packng s done n a best ft anner (.e., sallest budget porton s flled frst). Note that, segents wthn the sae phase have no precedence constrants and hence can be packed n any order. Furtherore, snce the executon of a segent s arbtrarly dvsble, t turns out that t s always possble to pack segents such that all budgets runnng at te t fnsh the executon of segents of phase sultaneously at that te. The only constrant to consder s that portons of the sae segent cannot run n parallel on ultple processors at the sae te. Hence, when schedulng a segent, the best ft polcy skps te ntervals where potons of the sae segent have already been scheduled. The exact pseudocode for the best ft segent packng algorth and the proof that t always succeeds at fndng a vald schedule between successve synchronzaton ponts are delegated to the appendx. Segents of Phase, apped by Applcaton Master (Packng Server) to budget slots whle respectng synchronzaton ponts t - Phase t Budget schedule coputed by Resource Manager (the underlyng scheduler ) Fgure 4: An exaple of segent packng. A. Ltatons The above dscusson has been a splfed treatent of MapReduce applcatons. MapReduce s a coplex syste. A fathful analytc treatent goes beyond the scope of a sngle paper. It s therefore useful to outlne the approxatons and splfcatons we ade n ths work. Frst and foreost, we do not explctly address data allocaton. Segents of task workflows operate on data. Such data ust be avalable on the local achne. If not, the coputaton te of the segent wll ncrease. In prncple,

8 CDF 0.5 Map Reduce Task Executon Te (s) Fgure 5: Yahoo! Hadoop Jobs Mapper and Reducer WCET t s possble to plan ahead of te, such that data are dstrbuted to achnes as segents are allocated, such that each segent fnds ts data locally. Challenges arse n the presence of preepton and graton, when a segent ght fnd tself resued on a dfferent achne. In general, ovng data around s a bad dea. Hence, n practce, the underlyng scheduler should consder graton and data oveent costs. For exaple, parttoned schedulng would be hghly preferable to global schedulng. The underlyng schedulng polcy s an orthogonal ssue to our contrbuton and hence s not addressed n ths paper. Second, the cost of preepton n MapReduce systes s hgher than that n a ult-core platfor. The NatJa syste [22] enables MapReduce preepton by nsertng checkponts between two key-value pars, and wrtng those checkponts nto the shared dstrbuted fle syste, whch ntroduces a few llseconds to a few seconds of delay. In real-world scenaros, ths s not a bg proble as MapReduce segents also take uch longer te to fnsh copared to ult-core tasks, leadng to a sall relatve preepton cost. For exaple, Fgure 5 plots the dstrbuton of apper and reducer WCET n Yahoo! Hadoop cluster [7]. WCETs of ost segents are larger than 0 seconds, and ore than 50% segents take ore than nute to fnsh. Fnally, there s a data oveent phase followng each coputatonal phase n a MapReduce workflow, where outputs of one set of segents are sent to the next set of segents. Ths oveent takes place over a hgh-speed network nterconnect. If network bandwdth s not suffcent, data oveent ay ntroduce delays that need to be explctly accounted for. One possblty s sply to subtract those delays fro end-to-end deadlnes, such that the deadlnes used reflect the te avalable for the coputatonal part only. Better solutons wll be explored n subsequent work. VI. EVALUATION In ths secton, we copare our soluton to two baselnes. The frst one s the state-of-the-art generalzed parallel tasks schedulng algorth [24], called federated schedulng. Aong past results, federated schedulng acheves the hghest-known utlzaton bound of 50% for generalzed parallel tasks, f the stretch surpasses 2. The second baselne for the parallel task odel s the GEDF schedulng 2 algorth, wth a best-known utlzaton bound of % [26]. A. Coputng the Optal Budget Sze We begn by coputng the optal Packng server budget sze (or equvalently, the optal value of ). Packng servers ay use any plct-deadlne ndependent task schedulng polcy A as the underlyng scheduler to schedule ther budgets, whle achevng a utlzaton bound U B for schedulablty of MapReduce workflows. In followng experents, we set A to EDF-FF [25] and (ndependent task) GEDF [26], respectvely, as they lead to hgh utlzaton bounds when stretch,, s large. We copute the optal as follows: EDF-FF: When EDF-FF s used as the underlyng polcy, A, the utlzaton bound guaranteed by Packng servers for MapReduce workflows becoes: U B ( +)( ). () ( +) By takng the dervatve wth respect to, and settng the dervatve to 0, the hghest utlzaton bound s acheved when: ( +)( ). (2) GEDF: When the underlyng schedulng polcy, A, s set to GEDF, the schedulablty bound becoes: U B ( )( +). (3) Slarly, by equatng the dervatve to zero, the optal, that acheves the hghest bound, s: ( ). (4) The followng experents use these two optal forulas to confgure Packng servers when they run above EDF-FF and GEDF schedulers. B. Schedulable Utlzaton The frst queston we answer n the evaluaton secton s to eprcally deterne the average schedulable utlzaton of a MapReduce cluster, due to tasks that eet deadlnes, under dfferent schedulng polces. Four polces are copared: () Packng servers on top of EDF-FF, () Packng servers on top of GEDF, () the federated schedulng polcy [24] (wth no Packng servers), and (v) GEDF. In ndustry, outputs of MapReduce workflows power a varety of servces, where tardness are usually allowed, but at the cost of dnshng onetary benefts. In our experents, we nhert the sae confguraton, allowng tasks to contnue executon after ther deadlnes, whch ay adversely affect the schedulablty of subsequent obs. We do not use adsson control. Rather, we vary the total nput workload utlzaton (as percentage of total platfor capacty) on the x-axs and count on the y-axs only the utlzaton of tasks whose deadlnes were

9 Accepted Utlzaton Packng & EDF FF 0.6 Packng & GEDF 0.5 Federated Packng & EDF FF 0.4 GEDF Packng & GEDF 0.3 GEDF Federated Subtted Utlzaton Accepted Utlzaton Packng & EDF FF 0.7 Packng & GEDF Federated Packng & EDF FF 0.4 GEDF Packng & GEDF 0.3 GEDF Federated Subtted Utlzaton Fgure 6: Schedulable Utlzaton, 20Fgure 7: Schedulable Utlzaton, PDF Workflow Response Te / Deadlne Fgure 8: Adsson Control et. Whle we do not explctly plot deadlne sses, note that the dfference between each curve and the dagonal xy s the utlzaton attrbuted to tasks that ssed deadlnes. In these experents, workflows are generated based on Yahoo! MapReduce cluster trace data [7, 8]. The Yahoo! dataset does not specfy the deadlne of obs or workflows. We therefore calculate the crtcal path length of workflows, and set ther deadlnes to control the value of. More specfcally, the nuber of slots s set to 500, and the paraeter s tuned to 20 (and 30) durng our sulatons, resultng n 3.58 (and 4.56) for the EDF-FF scheduler, and 4.47 (and 5.47) for the GEDF scheduler. Fgures 6-7 depct the experent results. The horzontal lnes n these fgures ndcate the theoretcal utlzaton bounds for the schedulablty of each of the four schees copared. When GEDF acts as the underlyng scheduler, the theoretcal schedulablty bounds of Packng servers are 60.3% for 20, and 66.9% for 30. Packng servers on top of EDF-FF have a 64% and 70% utlzaton bound, whch are the hghest under the 20 and 30 confguratons. We also plot the eprcally deterned, schedulable utlzaton curves for each of the four polces. Eprcally, when GEDF s the underlyng schedulng polcy, both GEDF (alone) and Packng servers on GEDF eet deadlnes of alost all tasks when the task set utlzaton s below 90%. However, the GEDF-based algorths fal serously, when the task set utlzaton surpasses 95%, exhbtng a dono effect. Federated schedulng leads to a theoretcal utlzaton bound of 50%, whch s the hghest-known bound n prevous work. Under MapReduce workloads, t s eprcally shown to be able to schedule tasks wthout deadlne sses up to about 70% utlzaton. Above that utlzaton deadlnes are ssed, although the dono effect seen by GEDF s not experenced. Packng servers on an EDF-FF scheduler appear to be the ost successful polcy. The polcy offers a hgh theoretcal schedulablty utlzaton bound, and perfors very well above the bound, rsng up to alost a 90% utlzaton wthout deadlne sses, when servng MapReduce workloads. No dono effect s experenced above 90%. Hence, we pleent ths scheduler on a 46-server Hadoop cluster to verfy ts feasblty and valdty. C. Meetng Deadlnes n a Real Hadoop Cluster Next, we test the effcacy of adsson control schees based on our new bounds at elnatng all deadlne sses. We pleent a prototype of Packng servers on WOHA [8], a workflow-enabled varant of Hadoop v. The experent runs on a cluster of 40 Dell PowerEdge R620 servers and 6 Dell PowerEdge R60 servers. The 40 R620 servers for a Hadoop cluster, provdng 60 reduce slots. The 6 R60 servers execute resource planner, and 5 clent nodes that subt workflow nvocatons to the Hadoop cluster. The paraeter s set to 20. It corresponds to a utlzaton bound of 64.3%, whch we set as the threshold for adsson control. Hence, we prepare a set of workflows wth a total utlzaton above 00%. An adsson controller s used that denes a workflow f t brngs the cluster utlzaton above 64.3%. All segents are coputatonallyntensve. Fgure 8 shows the resultng probablty densty dstrbuton of response te-to-deadlne rato of workflow nvocatons durng a 4-hour experent. All ratos are below, suggestng the valdty of the utlzaton bound. VII. RELATED WORK Workflow schedulng attracts ncreasng attenton fro both real-te and MapReduce researchers. The wdespread MapReduce deployents stulate the MapReduce county to desgn and prove schedulng polces for MapReduce pleentatons, such as Hadoop. The default scheduler executes obs n a FIFO order, leadng to poor farness under ult-tenant scenaros. Yahoo! developed a Capacty

10 Scheduler [20] to offer each Hadoop cluster tenant a guaranteed resource share. Facebook s Far Scheduler [2] organzes Hadoop obs nto pools, and farly dvdes resources aong these pools. Vera et al. [] evaluates an EDFbased schedulng algorth on MapReduce. Ther sulaton results confr that sple deadlne-based schedulng heurstcs allow ore obs to eet ther deadlnes. All of the above solutons target ob schedulng rather than workflow schedulng. Yahoo! later developed Ooze [9, 29] as a generc Hadoop workflow anageent tool, that subts each workflow ob at the rght te. WOHA [8] ntroduces deadlne-aware schedulng of Hadoop workflows. However, these schedulers ake no guarantees on whether workflow deadlnes are et or not. In real-te lterature, workflow schedulng (called generalzed parallel tasks schedulng) has been recently studed on ultprocessor platfors. Baruah et al. [30] prove that EDF can acheve a 2X speedup bound for a sngle recurrent workflow. Safullah et al. [3] propose to arrange a workflow nto stages, and then the workflow s deadlne s splt and assgned to each stage. If soe optal algorth can successfully schedule the orgnal workflow, ther soluton s guaranteed to satsfy the sae deadlne wth a 4X (speedup bound) speed processors. When the workflow s restrcted to a fork-on odel [32, 33], Lakshanan et al. [34] prove the speedup bound to L et al. [35] develop a capacty augentaton bound of 4 2 for workflows, whch edately leads to a sple and effectve schedulablty test. More recently, L et al. [24] prove the capacty augentaton bound to 2 usng the federated schedulng algorth. Nevertheless, a utlzaton below 50% ay be pessstc for ndustry MapReduce clusters. Independent tasks on ultprocessors have been studed ore extensvely durng the past decades. Soe algorths push the schedulablty bound to be uch hgher than 50%. The EDF-FF (frst ft) algorth s able to schedule all tasks f ther total utlzaton stays below U B + +, where s the nverse of the axu ndvdual task utlzaton (.e., u ax ). The global EDF guarantees ( schedulablty ) f the total utlzaton s less than U B +. Both algorths approach a 00% utlzaton bound when and are large, whch s coon on MapReduce platfors. These observatons otvate us to develop the Packng server technque to be able to apply those hgher bounds fro ndependent task schedulng to MapReduce workflows. VIII. CONCLUSION Ths paper ntroduces the technque of Packng server to convert ndependent task set schedulablty bounds to MapReduce workflows schedulablty bounds. If an ndependent task set scheduler A guarantees schedulablty up to total utlzaton U B, the Packng servers can acheve schedulablty bound of U B usng A as the underlyng scheduler, where s the nu deadlne to crtcal path rato, and [,] s a tunable paraeter that curbs the axu converted ndvdual ndependent task utlzaton (u ax ). MapReduce workflows usually yeld large, allowng the new bound to acheve a uch hgher value than the best known bound of 50%. Our evaluatons usng Yahoo! data on a 46-server Hadoop cluster confr the valdty of the new bound and the feasblty of the Packng server syste desgn. ACKNOWLEDGEMENT Ths work was sponsored n part by the Natonal Scence Foundaton under grants CNS , CNS , CNS and CNS We are also thankful to Yahoo! Inc. for sharng the access to Webscope data. REFERENCES [] J. Dean and S. Gheawat, Mapreduce: Splfed data processng on large clusters, n USENIX OSDI, [2] Apache Hadoop, October 204. [3] Apache Hadoop V2 (YARN): Yet Another Resource Negotator, October 204. [4] Apache spark, May 204. [5] S. L, T. Abdelzaher, and M. Yuan, Tapa: Teperature aware power allocaton n data center wth ap-reduce, n IEEE IGCC, 20. [6] H. K, D. de Nz, B. Andersson, M. Klen, O. Mutlu, and R. Rakuar, Boundng eory nterference delay n cotsbased ult-core systes, n IEEE RTAS, 204. [7] J. Lee, A. Easwaran, and I. Shn, Maxzng contentonfree executons n ultprocessor schedulng, n IEEE RTAS, 20. [8] C. Lu and J. H. Anderson, Suspenson-aware analyss for hard real-te ultprocessor schedulng, n ECRTS, 203. [9] M. A. Haque, H. Aydn, and D. Zhu, Real-te schedulng under fault bursts wth ultple recovery strategy, n IEEE RTAS, 204. [0] D. Logothets, C. Trezzo, K. C. Webb, and K. Yocu, In-stu apreduce for log processng, n USENIX ATC, 20. [] A. Vera, L. Cherkasova, V. S. Kuar, and R. H. Capbell, Deadlne-based workload anageent for apreduce envronents: Peces of the perforance puzzle, n IEEE NOMS, 202. [2] K. Kc and K. Anyanwu, Schedulng hadoop obs to eet deadlnes, n IEEE CLOUDCOM 200. [3] K. Agrawal, C. Gll, J. L, M. Mahadevan, D. Ferry, and C. Lu, A real-te schedulng servce for parallel tasks, n IEEE RTAS, 204. [4] R. M. Pathan, P. Stenstró, L.-G. Green, T. Hult, and P. Sandn, Overhead-aware teporal parttonng on ultcore processors, n IEEE RTAS, 204. [5] J. Forget, F. Bonol, E. Grolleau, D. Lesens, and C. Pagett, Schedulng dependent perodc tasks wthout synchronzaton echanss, n IEEE RTAS, 200. [6] C. J. Kenna, J. L. Heran, B. C. Ward, and J. H. Anderson, Makng shared caches ore predctable on ultcore platfors, n ECRTS, 203. [7] Yahoo! webscope, [8] S. L, S. Hu, S. Wang, L. Su, T. Abdelzaher, I. Gupta, and R. Pace, Woha: Deadlne-aware ap-reduce workflow schedulng fraework over hadoop cluster, n IEEE ICDCS, 204.

11 [9] Apache ooze, May 204. [20] Capacty scheduler, capacty scheduler.htl, May 204. [2] Far scheduler, scheduler.htl, May 204. [22] B. Cho, M. Rahan, T. Chaed, I. Gupta, C. Abad, N. Roberts, and P. Ln, Nata: Desgn and evaluaton of evcton polces for supportng prortes and deadlnes n apreduce clusters, n ACM SoCC, 203. [23] S. K. Baruah, V. Bonfac, A. Marchett-Spaccaela, L. Stouge, and A. Wese, A generalzed parallel task odel for recurrent real-te processes, n RTSS, 202. [24] J. L, J. J. Chen, K. Agrawal, C. Lu, C. Gll, and A. Safullah, Analyss of federated and global schedulng for parallel realte tasks, n ECRTS, 204. [25] J. M. López, M. García, J. L. Díaz, and D. F. García, Worst-case utlzaton bound for edf schedulng on real-te ultprocessor systes, n ECRTS, [26] T. P. Baker, A coparson of global and parttoned edf schedulablty tests for ultprocessors, In Internatonal Conf. on Real-Te and Network Systes, Tech. Rep., [27] J. Wong, Whch bg data copany has the worlds bggest hadoop cluster? Septeber [28] G. Lpar and E. Bn, A fraework for herarchcal schedulng on ultprocessors: Fro applcaton requreents to runte allocaton, n IEEE RTSS, 200. [29] M. Isla, A. K. Huang, M. Battsha, M. Chang, S. Srnvasan, C. Peters, A. Neuann, and A. Abdelnur, Ooze: towards a scalable workflow anageent syste for hadoop, n ACM SWEET, 202. [30] S. Baruah, V. Bonfac, A. Marchett-Spaccaela, L. Stouge, and A. Wese, A generalzed parallel task odel for recurrent real-te processes, n IEEE RTSS, 202. [3] A. Safullah, K. Agrawal, C. Lu, and C. Gll, Mult-core real-te schedulng for generalzed parallel task odels, n IEEE RTSS, 20. [32] C. Maa, L. Noguera, L. M. Pnho, and M. Bertogna., Response-te analyss of fork/on tasks n ultprocessor systes, n ECRTS, 203. [33] C. Maa, M. Bertogna, L. Noguera, and L. M. Pnho, Response-te analyss of synchronous parallel tasks n ultprocessor systes, n Proceedngs of the 22Nd Internatonal Conference on Real-Te Networks and Systes (RTNS), 204. [34] K. Lakshanan, S. Kato, and R. Rakuar, Schedulng parallel real-te tasks on ult-core processors, n IEEE RTSS, 200. [35] J. L, K. Agrawal, C. Lu, and C. Gll, Analyss of global edf for parallel tasks, n IEEE ECRTS, 203. APPENDIX A. Schedulng Segents on Budgets As Algorth converts MapReduce workflows nto MapReduce ppelnes wthout ntroducng any utlzaton penalty, we dscuss the schedulng algorth n the context of a MapReduce ppelne for sake of splcty. In order to further splfy the notaton, we focus on a sngle MapReduce ppelne nvocaton fro MapReduce task τ, savng subscrpts for task ID and ppelne ID. The algorth schedules each phase nto ts budget portons n a Frst-Ft anner. Startng fro the frst phase, let π {π,π 2,..., π z } denote the set of budget portons for the frst phase, where z. Please note, soe budget ay copletely fall n te nterval [t, + ), leavng z to be saller than. A budget porton π s a set of reserved non-overlappng te ntervals on soe resource slots. Hence, each budget porton assocates wth two affntes, te and slot. Let N (π) represent the nuber of budget portons n π (.e., N (π) z), and L(π) the total sze of budget portons n π (.e., L(π) L(π )). Wthout loss of generalty, assue the set π s ordered such that L(π ) L(π + ).Astwo budgets cannot execute on the sae slot at the sae te, we have L(π u \π v )L(π u ) and L (π u π v )L(π u )+L(π v ), for u v. Algorth 2 schedules x segents of WCET y on the budget porton set π. Algorth 2 Schedule a phase on ts budget portons Input: π the set of budget portons, x the nuber of segents, y the length of each segent Output: S the schedule of nput segents usng n nput budget portons : procedure SCHEDSEG(π, x, y) 2: S 3: for x do 4: l y 5: for N(π) do 6: Schedule the length-l segent on π followng the ncreasng order of te, skppng all conflctng te nstances. 7: Store the scheduled ( ) part n π 8: l l L π 9: S S {π } 0: end for : f l>0 then 2: return null 3: end f 4: end for 5: return S 6: end procedure The algorth schedules all segents one by one (Lnes 3-4). For each segent, t always tres to use saller budget portons frst (Lnes 5-0). A segent flls budget porton π followng the ncreasng order of te. Reanng parts n π wll be flled by followng segents. In order to prevent segent-level parallels, the algorth skps all conflctng te ntervals when schedulng segents. As Algorth 2 only requres a budget porton set and segents propertes, t also apples to the subsequent phase n the ppelne, by settng x to ax{, }, y the WCET c, and π the budget portons n te nterval [t,t ). B. Algorth Correctness We now prove that Algorth 2 guarantees to successfully schedule a phase on correspondng budget portons. Agan, n order to splfy the notaton, we focus on one sngle MapReduce ppelne fro MapReduce task τ, and use the frst phase as the proof subect, savng the notatons for task ID, and phase ID. The proof can be dvded nto two cases dependng on whether the sze of the sallest budget porton L(π ) s larger than segent WCET y.

12 Case : L(π ) >y. Algorth 2 always tres to fll up budget porton π before t starts to use π +, unless segent-level parallels conflcts prevent t fro achevng that. In the case of L(π ) >y, a segent α can ether copletely ft nto the current budget porton π, or exhaust the reanng parts of the π and start to use π +.Asπ + π >y, the unscheduled parts of α can always ft nto π + avodng parallels conflcts fro π. Therefore, budget portons are flled up one-by-one followng ther ndex order, leavng no gap n the ddle. Due to L(π) xy, all segents can be scheduled n π. Case 2:L(π ) y. We apply nducton on z. Bass: When z, all segents are scheduled sequentally nto a sngle budget porton. The nducton hypothess trvally holds. Inductve Step: Assue the lea holds for z k, we now prove t also holds for z k. There are two cases: As Algorth 2 s deternstc, t s easy to fgure out whch parts of π are assgned to the frst segent. Reove those parts fro π, and denote the resultng budget porton set as π. Now, n order to prove the lea, we only need to show that Algorth 2 s able to ft the reanng x length-y segents nto schedule π.gvenn (π) x, L(π ) y, and L(π) xy, wehaven (π ) x and L (π )L (π) y (x )y. Therefore, accordng to the nducton hypothess, Algorth 2 can ft x length-y segents nto π, plyng that the lea also holds for z k.