Scheduling Hadoop Jobs to Meet Deadlines

Transcription

1 Scheduling Hadoop Jobs to Meet Deadlines Kamal Kc, Kemafo Anyanwu Depatment of Compute Science Noth Caolina State Univesity Abstact Use constaints such as deadlines ae impotant equiements that ae not consideed by existing cloud-based data pocessing envionments such as Hadoop. In the cuent implementation, jobs ae scheduled in FIFO ode by default with options fo othe pioity based schedules. In this pape, we extend eal time cluste scheduling appoach to account fo the two-phase computation style of MapReduce. We develop citeia fo scheduling jobs based on use specified deadline constaints and discuss ou implementation and peliminay evaluation of a Deadline Constaint Schedule fo Hadoop which ensues that only jobs whose deadlines can be met ae scheduled fo execution. I. INTRODUCTION Apache Hadoop is an open souce implementation of Google s MapReduce [4] that has gained significant populaity as a platfom fo lage scale data pocessing applications. MapReduce is a paallel data pocessing paadigm tageted at cluste-based computing achitectues. Its advantage is that it allows pogammes to abstact fom the issues of paallelization, scheduling, input patitioning, failove, eplication and focus on designing thei application data flows consisting of filteing and aggegation steps. The MapReduce pogamming model consists of encoding data pocessing in tems of two functions: Map and Reduce. Input data is patitioned into fixed sized blocks and fed into paallel Map tasks which pocess the data chunks and poduce intemediate output as a collection of key-value pai tuples. These tuples ae shuffled acoss diffeent educe nodes based on key values. Each Reduce task pefoms thee steps: copy - the map output is copied to educe nodes, sot - the collected map output is soted based on key values and educe - educe function e.g. aggegation is applied to the data. Vaious effots like Hive [5], Pig[11] offe fiendlie intefaces in the fom of high level quey o dataflow languages a la SQL. This enables uses to encode thei tasks in tems of quey opeatos that ae automatically compiled into Hadoop jobs (MapReduce wokflows) instead of as low-level Map and Reduce functions. The MapReduce achitectue consists of one maste (Jobtacke) and many wokes (Tasktackes). The JobTacke eceives job submitted fom use, beaks it down into map and educe tasks, assigns the tasks to Tasktackes, monitos the pogess of the Tasktackes, and finally when all the tasks ae complete, epots the use about the job completion. Each Tasktacke has a fixed numbe of map and educe task slots that detemine how many map and educe tasks it can un at a time. The Hadoop File System HDFS suppots eliability and fault toleance of MapReduce computation by stoing and eplicating the inputs and outputs of a Hadoop job. Since Hadoop jobs have to shae the cluste esouces, a scheduling policy is used to detemine when a job can execute its tasks. The default scheduling policy of Hadoop is Fist In Fist Out (FIFO). Unde this scheme, the job that was submitted ealie gets pefeence ove jobs submitted late. Recent effots such as Delay Schedule[15], Dynamic Popotional Schedule [13] offe diffeentiated sevice fo Hadoop jobs allowing uses to adjust the pioity levels assigned to thei jobs. Howeve, this does not guaantee that the job will be completed by a specific deadline. [12] comes close to addessing the issue of deadlines but focuses moe on inceasing system utilization. [14], [9] focus on suppoting deadline constaints in taditional paallel computatioodels which diffe fom the two-phase computation and unique dataflow of MapReduce jobs. [1] consides deadline constaints in the context of eal time tansactions in single pocesso envionments. In this pape, we lay the foundation fo dealing with deadline equiements in Hadoop-based data pocessing by (1) poposing a job execution cost model that accounts fo the vaious paametes that affect Hadoop job completion time such as map and educe untimes, map and educe input data sizes, data distibution, etc., (2) pesenting the design of a Constaint-Based Hadoop Schedule that takes use deadlines as pat of its input and detemines the schedulability of a job based on the poposed job execution cost model and does so independent of the numbe of jobs unning in the cluste. Jobs ae only scheduled if specified deadlines can be met. We focus on deadline constaints when MapReduce untime paamete values ae known and leave the issue of estimating job paamete values as futue wok. The est of the pape is oganized as follows: in section II, we discuss the scheduling aspects of the poblem and deive expession fo minimum map/educe task allocation equied to meet deadlines. In section III, we pesent the design and implementation of Constaint Schedule and in section IV, we pesent esults fo task allocation fo diffeent deadlines. A. Poblem Definition II. FOUNDATIONS Poblem Statement: Can a given quey q that tanslates to a MapReduce job J and has to pocess data of size σ be completed within a deadline D, when un in a MapReduce cluste having N nodes with N m map task slots, N educe task slots and possibly k jobs executing at the time. Executing quey q in the MapReduce famewok involves scheduling the coesponding map and educe tasks of job J.

2 While it is possible fo q to tanslate to a sequence of MapReduce jobs, in this pape, we focus on queies that tanslate to a single MapReduce job. Afte a job is submitted, the schedule fist needs to detemine whethe the job can be completed within the specified deadline o not using a schedulability test. Rathe thaake schedulability detemination based on all the jobs unning in the system, we focus on the fee slots availability at the given time o in the futue. Afte it is detemined that the job can be completed within the given deadline, it is enlisted fo scheduling. An impotant issue is then to assign the ight numbe of tasks to the Tasktacke to ensue that the deadline is met. Some stategies include: Assign all map and educe tasks: Assigns all the tasks if the numbe of tasks is less than the total available slots. If not, it assigns the tasks to all the available slots in the cluste. This may esult in jobs submitted late not having enough slots to un. Assiginimum tasks: Assigns only minimum numbe of tasks equied fo a job to meet its deadline. Empty slots may be available fo jobs submitted late. Assign some fixed numbe of tasks. Fo the est of pape, minimum numbe of tasks will be used to efe to the minimum numbe of tasks equied fo a job s schedulability independent of task assignment appoach. A job is schedulable if the minimum numbe of tasks fo both map and educe is less than o equal to the available slots. We show the deivation fo the minimum numbe of map and educe task in the following subsections. B. Deadline Estimation Model We develop an initial estimatioodel based a set of assumptions which will be elaxed late in the discussion: (1) the cluste consists of homogeneous nodes, so that the unit cost of pocessing fo each map o educe node is equal; (2) key distibution of the input data is unifom, so that each educe node gets equal amount of educe data to pocess; (3) educe tasks stats afte all map tasks have completed; (4) the input data is aleady available in HDFS. To deive the expessions fo the minimum numbe of map tasks in m and educe tasks in, we extend the model used in[9] fo Equal Load Patitioning technique by intoducing MapReduce specific notations J, f, c m, c, s m and s as descibed below: q = (A, σ, D): A quey q, whee A is the aival time (time when the quey is submitted), σ is the input data size, D is the elative deadline. J = (t m1, t m2, t m3,..., t mu, t 1, t 2,..., t v ): A Hadoop job that is un to pefom quey q. t mi is the i th map task and t j is the j th educe task whee 1 i u and 1 j v. The Hadoop job J has the aival time, input data and deadline same as the quey q = (A, σ, D). n: total slots assigned to the job in the cluste. n = + n whee, is the map slots and n is the educe slots. α = (α 1, α 2,..., α u ): Map data distibution vecto, whee u is the total numbe of map tasks of job J. α i is the data faction allocated to the i th map task. Map input data is equally distibuted among the map nodes so, α i = 1 u. f : Filte atio. The faction of input that the map pocess poduces as output. Fo most pactical puposes 0 f 1 holds. f σ: Reduce input (map output) = poduct of filteatio and the map input. c m : Cost of pocessing a unit data iap task. c : Cost of pocessing a unit data in educe task. c d : Communication cost of tansfeing unit data. s m : stat time of the fist map task fo the job s : stat time of the fist educe task fo the job To estimate the duation of the job J we conside map completion time, educe completion time and data tansfe duing educe copy phase. The expession fo the cost can be witten as: σc m + fσc n + fσc d Since the job has an aival time A and deadline D, Let, s max then, which gives, theefoe, Similaly, s m + σcm + fσc n + fσc d A + D be the maximum value fo the educe stat time, s max in = = A + D fσc n s m + σcm in m = s max σcm s max s m s max fσc d σc m s m fσc A + D fσc d s Ou constaint schedule uses this citeia to schedule Hadoop jobs. 1) Invalidating Assumptions: Assumption 1. We assumed that nodes ae homogeneous and the value of c m and c was unifom fo all the nodes. To elax this assumption we allow fo the value of c m and c to diffe among the nodes making the job dependent on the slowest node i.e. the value of c m and c is the highest. Theefoe, we modify the above expessions fo in m, in and s max by substituting c m and c with the value of the slowest unning node. Assumption 2: When the key distibution acoss the educe tasks is not unifom, then the wost case is that a single node pocesses all the educe input fσ. In such scenaio, the value of s max will be: s max = A + D fσc fσc d We intend to addess in ou futue wok the effect of vaious data distibutioodels. Stating educe tasks ealy might affect othe jobs ability to obtain educe task slots. In (1) (2)

3 this pape, we do not addess the implications of invalidating Assumption 3. We think it is elevant to assume that the input is available in HDFS because MapReduce pocessing stats only afte the data is available. If it becomes impotant to conside the cost of tansfeing the input to HDFS fom use s stoage, the cost can be deived using the values fo tansfe netwok link capacity and the eplication cost, and these costs ae not dependent on the natue of MapReduce job. III. DESIGN AND IMPLEMENTATION Hadoop suppots pluggable schedules and we have implemented Constaint Schedule using the minimum task scheduling citeia developed in the pevious section. It is developed as a contib module using Hadoop vesion souce code. Hadoop config file needs to be modified to use the schedule. We have also implemented a web based inteface that allows the use to specify the deadline fo a given job. A. Design Goals The design goals fo Constaint Schedule wee: (1) To be able to give uses immediate feedback on whethe the job can be completed within the given deadline o not and poceed with execution if deadline can be met. Othewise, uses have the option to esubmit with modified deadline equiements. (2) Maximize the numbe of jobs that can be un in the cluste while satisfying the time equiements of all jobs. When a job is submitted, we pefom its schedulability test simila to that mentioned in [9]. We fist calculate the minimum map tasks fo the submitted job. If the minimum map tasks ae not available at the time when job was submitted based on the total numbe of educe tasks specified fo the job. As mentioned ealie, the numbe of educe tasks fo a job is use defined o is a default value. If the numbe of educe task then the job is ejected. We then calculate s max slots available at s max is not equal to the specified numbe of educe tasks fo the job then also the job is ejected. An altenative to ejecting a job stictly based on s max would be to compute a suitable s value so that the equied numbe of educe tasks slots would be less than the job specified numbe. We intend to exploe this aspect in ou futue wok. Afte a job is scheduled and map tasks ae complete, in is computed to detemine how many educe tasks should be scheduled. Since, s s max, this allows fo cases when in can be less than the educe tasks specified fo the job. This inceases the potential of keeping some educe slots empty. Anothe design goal is to maximize the numbe of jobs while satisfying the deadlines. Thee have been aguments fo and against it when using the minimum tasks scheduling appoach fo multipocesso and cluste computing envionment as mentioned in [8],[9]. Aival ates of the job may play an impotant ole in detemining this. Intuitively, when jobs ae scheduled using only minimum numbe of tasks then it leaves slots empty fo late jobs to execute. We, howeve do not quantitatively evaluate it in this pape. Algoithm 1 scheduletasks(tasktacke t) assignedt asks.initialize() f eemapslots t.f eem apslots f eeeduceslots t.f eereduceslots epeat j nextjob(p ioityqueue) if!j.mapf inished AND j.scheduledm aps < j.minm apt asks then tasks j.getm apt asks(f eemapslots) f eemapslot f eemapslot tasks.size assignedt asks.append(tasks) end if if j.mapf inished AND j.scheduledreduce < j.minreducet asks then tasks j.getreducet asks(f eeeduceslots) f eeeduceslots f eeeduceslots tasks.size assignedt asks.append(tasks) end if until endof Queue(P ioityqueue) OR (feemaptasks == 0 and feeeducetasks == 0) etun assignedt asks B. Task Assignment and Runtime Map and educe tasks untime values ae equied to deive the minimum node citeia. In ou implementation we use static values. A use can supply the map and educe task untime values using the web inteface. In ou futue wok we intend to exploe techniques fo task untime estimation such as cost model fo MapReduce code and estimating cost using a shot sampled job. Recent wok such as Manimal [3] have exploed static code analysis techniques fo MapReduce code. Task assignment is done in eply to the heatbeat obtained fom the Tasktackes. The heatbeat inteval is 3 seconds by default. Constaint Schedule maintains a pioity queue of the jobs odeed by thei deadlines. Tasks scheduling is fist attempted fo jobs that is at the font of pioity queue. If a job s minimum task count is satisfied then we conside next job until thee is no moe jobs emaining in pioity queue o until thee ae no moe unassigned map/educe slots emaining fo the Tasktacke. Thee can be Tasktackes which may not be assigned any tasks if the minimum task count is satisfied fo all the jobs. Algoithm 1 pesents this technique fo scheduling tasks. The scheduling algoithm assigns educe tasks only afte all the map tasks have been completed. This is in accodance with ou assumption in pevious section. IV. EVALUATION To evaluate the Constaint Schedule we an Hadoop job that epesented aggegation opeation, which is one of the common type of opeation pefomed by MapReduce. MapReduce job equivalent to the following quey was used fo the expeiment. SELECT useid, COUNT(actionid) AS num_actions FROM USERACTION; The USERACTION table contains tuples (useid,actionid). The map task of the job

4 pases the input and outputs (useid, actionid) as key-value pais. The educe task then counts the numbe of actionid fo each useid. The filteatio fo this MapReduce job is 1. The geneated data used fo the expeiment had unifom distibution acoss the key - useid, esulting in educes getting equal amount of data to pocess. We have not consideed map and educe task statup ovehead in ou expeiments. We also do not conside the failue ate of the cluste in the expeiments. Data tansfe time was estimated using ipef [6] by pefoming communication between all Jobtacke and Tasktackes. We evaluate the task assignment behavio of Constaint Schedule fo diffeent deadlines. Ou obsevations ae descibed below. A. Expeimental setup Expeiments wee conducted in two diffeent cluste envionments: vitualized cluste and physical cluste. The vitualized cluste consisted of a single physical node with 3 guests as Tasktackes and the host system as Jobtacke. The machine had 4 GB memoy with 64 bit Intel dual coe 2.93GHz pocesso and Ubuntu seve pesent in each vitualized nodes. The physical cluste was a 11 node cluste eseved using VCL [7] whee 10 wee Tasktackes and 1 was Jobtacke. Each node specification was: 4GB maiemoy, Intel 2.33 GHz pocesso unning Redhat Entepise Linux. Hadoop vesion was used in the vitualized cluste and vesion was used in the physical cluste. Both the envionments had Tasktackes with 2 map slots and 2 educe slots. Othe than changing the schedule, all othe configuation paametes wee default values. The HDFS block size was 64 MB. The measued netwok capacity was 12MB/s. B. Results In the vitualized nodes, we obseved that the task execution time vaied depending upon how many nodes wee cuently executing the map tasks. This was due to cpu shaing between the vitualized nodes. Vitualization is one of the scenaios when heteogeneity exists among nodes. We accounted fo heteogeneity in poblem fomulation section by using the execution time of the slowest node in ou minimum task citeia. We obseved the task execution time vaying fom 37 seconds to 120 seconds. We selected the lagest execution time as ou estimate. Figue 1 shows the map task allocation esults fo the same job when submitted with diffeent deadlines. The input size was 975MB which esulted in 16 map tasks. The cluste capacity was 6 map tasks and 6 educe tasks. We obseve that the tasks allocation is done diffeently fo the two deadlines. Fo 600s deadline, 6 map tasks is scheduled at a time. Fo 700s deadline, 5 map tasks ae scheduled. The numbe of map tasks scheduled is the minimum tasks value deived by the schedule. Inteestingly, we also notice that the map tasks fo deadline 700s finishes ealie than the map tasks fo deadline 600s. This is due to the inceased map completion time fo deadline 600s, as it ties to schedule moe map tasks and that esults in inceased pe map completion time though it is within the initial estimate we povide. The incease in pe map #tasks d=600s map count d=700s map count time(ms) Figue 1. Task allocation fo jobs with deadlines 600s and 700s in cluste with vitualized hosts #tasks d=1000s map count d=1000s educe count d=680s map count d=680s educe count e+06 time(ms) Figue 2. Map and educe task allocation fo jobs with deadlines 1000s and 680s in cluste with physical hosts completion time is due to the cpu shaing among vitualized nodes as descibed peviously. Job with deadline 700s ties to schedule less map tasks which esults in bette map completion time and hence it finishes faste. The above esult was obtained by executing the jobs when the cluste had no othe unning jobs. The tansient spikes seen in Figue 1 coespond to the time inteval duing which a task has finished but the task s assigned Tasktacke o any othe Tasktacke with empty slot has not sent heatbeat equest. Similaly, Figue 2 shows the task allocation esults fo the same job unning on the physical cluste with deadlines of 1000s and 680s. The input size was 2.9 GB which esulted in 48 map tasks. The cluste capacity was 20 map tasks and 20 educe tasks. Fo 680s deadline, 20 map tasks and 5 educe tasks ae scheduled. Fo 1000s deadline, 8 map tasks and 4 educe tasks ae scheduled. In both the cases, the deadlines ae met and Constaint Schedule ensues that the minimum task count is met duing the entie job execution. To meet the minimum task citeia, a task may be assigned to Tasktacke even though the data is non local to it. We have not accounted this in ou minimum tasks citeia. Since,

5 in these expeiments the task count is less and also that we have used the obseved wost case computation time, the data tansfe cost due to non local computation seems to be hidden. Howeve, it s impact in scenaios involving lage numbe of tasks emains to be investigated. Also, ou basic model includes the cost of copy phase of educes but does not sepaately account fo sot phase. Duing the expeiment, we have assumed that the educe cost also includes the sot cost. V. RELATED WORK MapReduce scheduling and time estimation: Until ecently, options fo scheduling tasks in Hadoop wee limited to the default FIFO schedule, FaiSchedule [15] and Capacity Schedule. Latte two and ecent eseach effot Dynamic Popotional Schedule [13] povide moe job shaing and pioitization capability in scheduling esulting in inceased shaing of cluste esouces and moe diffeentiation in sevice levels of diffeent jobs. Time estimation and optimization fo Hadoop jobs has been exploed by [2],[10]. [2] focuses on minimizing the total completion time of a set of MapReduce jobs. [10] estimates the pogess of queies that un as MapReduce DAGs. Most effots on scheduling focus on handling vaious pioity and most time estimation effots ae focused on untime estimation of aleady unning jobs. The closest effot to ou wok is [12] which poposes a schedule that ties to incease system esouce utilization by attempting to hono time constaints simila to a deadline schedule. Howeve, they do not conside the schedulability of a job pio to accepting it fo execution. Also, thei wok emphasizes map tasks and does not model the educe computation. Ou wok attempts to cove these aspects and puts moe emphasis oeeting deadlines in the shaed cluste envionment. Scheduling database tansactions: In [1], the authos evaluate tansactions that have eal time constaints fo single pocesso memoy esident database systems. The authos use estimated execution time fo detemining the Feasible Deadline which detemines whethe a deadline will be met o not. Fo scheduling, they use Ealie Deadline Fist (EDF) and Least Slack method. Ou wok builds aound obtaining the execution estimate (in ou case estimate of map and educe computation). Scheduling unde constaints: A genetic algoithm fo task assignment in gid-based envionments is used in [14]. The scheduling poblem hee is the mapping of tasks onto a suitable sevice level to minimize the execution time of a wokflow and complete it within a given budget. [9] exploes the scheduling of divisible eal time tasks in a cluste envionment. A divisible task efes to a task that can be divided into multiple independent subtasks each of which will pocess a piece of input data. Its data patitioning using Equal Patitioning Rule and task/subtask concept can be consideed to be analogous to Hadoop. The authos deive an expession fo the minimum numbe of nodes equied to meet a deadline. Though similaities with ou wok exist in task assignments and data patitioning, thei wok assumes a single type of computation wheeas Hadoop has two types of computations: map and educe. Map input data size can be detemined when a job is submitted but the educe input size is not known without consideing the key distibution. Also, map computations ae almost unifom in homogeneous envionment but educe computation vay depending upon the size of input a educe task eceives. Consequently, ou wok extends thei basic deadline detemination citeia to account fo the MapReduce style of computation. VI. CONCLUSION In this pape, we extended the eal time cluste scheduling appoach to deive minimum map and educe task count citeia fo pefoming task scheduling with deadline constaints in Hadoop. We pesented the design and implementation of Constaint Schedule fo Hadoop following the poposed appoach. Ou esults show that when deadlines fo job is diffeent, then the schedule assigns diffeent numbe of tasks to Tasktacke and makes sue that the specified deadline is met. In this wok, we have left out aspects of Constaint Schedule such as map/educe task untime estimation, filteatio estimation, data distibution and multiple MapReduce cycle suppot. We plan to addess these topics in ou futue wok. REFERENCES [1] Robet K. Abbott and Hecto Gacia-Molina. Scheduling eal-time tansactions: a pefomance evaluation. ACM Tans. Database Syst., 17(3): , [2] Ashaf Aboulnaga, Ziyu Wang, and Zi Ye Zhang. Packing the most onto you cloud. In CloudDB 09: Poceeding of the fist intenational wokshop on Cloud data management, pages 25 28, New Yok, NY, USA, ACM. [3] Michael J. Cafaella and Chistophe Ré. Manimal: Relational optimization fo data-intensive pogams. In WebDB, [4] Jeffey Dean and Sanjay Ghemawat. Mapeduce: simplified data pocessing on lage clustes. In OSDI 04: Poceedings of the 6th confeence on Symposium on Opeating Systems Design & Implementation, pages 10 10, Bekeley, CA, USA, USENIX Association. [5] Hive. [6] Ipef. [7] Vitual Computing Lab. [8] Wan Yeon Lee, Sung Je Hong, and Jong Kim. On-line scheduling of scalable eal-time tasks oultipocesso systems. J. Paallel Distib. Comput., 63(12): , [9] Xuan Lin, Ying Lu, J. Deogun, and S. Goddad. Real-time divisible load scheduling fo cluste computing. In Real Time and Embedded Technology and Applications Symposium, RTAS th IEEE, pages , [10] Kisti Moton, Magdalena Balazinska, and Dan Gossman. Paatime: a pogess indicato fo mapeduce dags. In SIGMOD 10: Poceedings of the 2010 intenational confeence on Management of data, pages , New Yok, NY, USA, ACM. [11] Pig. [12] Joda Polo, David Caea, Yolanda Becea, Malgozata Steinde, and Ian Whalley. Pefomance-diven task co-scheduling fo mapeduce envionments. In Netwok Opeations and Management Symposium (NOMS), 2010 IEEE, pages , [13] Thomas Sandholm and Kevin Lai. Dynamic popotional shae scheduling in hadoop. In JSSPP 10: 15th Wokshop on Job Scheduling Stategies fo Paallel Pocessing, [14] Jia Yu and Rajkuma Buyya. A budget constained scheduling of wokflow applications on utility gids using genetic algoithms. In Wokshop on Wokflows in Suppot of Lage-Scale Science, Poceedings of the 15th IEEE Intenational Symposium on High Pefomance Distibuted Computing (HPDC. IEEE, IEEE CS Pess, [15] Matei Zahaia, Dhuba Bothaku, Joydeep Sen Sama, Khaled Elmeleegy, Scott Shenke, and Ion Stoica. Delay scheduling: a simple technique fo achieving locality and fainess in cluste scheduling. In EuoSys 10: Poceedings of the 5th Euopean confeence on Compute systems, pages , New Yok, NY, USA, ACM.