An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds *

JOURNAL OF INFORMATION SCIENCE AND ENGINEERING 30, 1537-1552 (2014) An Integrated Dynamc Resource Schedulng Framework n On-Demand Clouds * College of Computer Scence and Technology Zhejang Unversty Hangzhou, 310027 P.R. Chna E-mal: {lexu; zjuzhwang; chenwz}@zju.edu.cn The bggest advantage of employng vrtualzaton n cloud s the ablty to provson resource flexbly whch makes pay-as-use model possble. However, snce the workload of vrtual machne constantly changes, t s stll a challenge that effcently schedule resource by mgratng vrtual machnes among lots of hosts, especally wth mult-objectve to meet, lke power and QoS constrants. In ths paper, we present a dynamc resource schedulng soluton, named Smart-DRS, whch can well determnes when to schedule, whch to schedule, and where to schedule. It s an ntegrated framework that ams to deal well wth hotspot mtgaton, load balancng and server consoldaton whch are classc problems to solve n on-demand clouds. The expermental results show that our framework strkes a balance between effcency, overhead and nstantanety. Keywords: cloud, dynamc resource schedule, hotspot mtgaton, load balancng, server consoldaton 1. INTRODUCTION The ablty of vrtualzaton technology whch s an mportant enabler for cloud computng brngs mmense benefts n terms of relablty, effcency and scalablty. Vrtualzaton, coupled wth VM (Vrtual Machne) mgraton capablty, enables the cloud data centers to flexbly provson resource whch makes the computng-as-a-servce vson of on-demand clouds possble. Ths pay-as-use model enables user pay only for the resources they really used n whch compute resources are made avalable as a utlty servce an lluson of avalablty of as much resources (e.g., CPU, memory, and network bandwdth) as demanded by the user. It s the most mportant feature of IaaS (Infrastructure as a Servce) and also the bggest push of cloud computng. However, mplementng such an on-demand cloud data center requres an excellent DRS (Dynamc Resource Scheduler), a power and QoS aware scheduler. As we know, the energy consumpton of datacenter s an ncreasngly sharp problem. So we need to power off several PMs (Physcal Machnes) by consoldatng all VMs together to certan selected PMs when the workload s low. Meanwhle, to keep hgh QoS performance of cloud servce, we need to well manage resource to automatcally balance load and mtgate hgh load machne n tme. For achevng aforementoned goals, n ths paper, we present a three-step dynamc resource schedulng strategy, named Smart-DRS, whch s an ntegrated schedulng solu- Receved December 14, 2012; revsed November 8, 2013; accepted November 16, 2013. Communcated by Ruay-Shung Chang. * Ths work has been presented n 2012 Internatonal Workshop on Power and QoS Aware Computng (P- QoSCom 12) n conjuncton wth IEEE Cluster 12, Sept. 2012, Bejng, sponsored by IEEE, IEEE Computer Socety, IEEE TCSC, Intel, VMware, NVda, etc. 1537

1538 ton that proved to have a good performance by expermentng n the prototype system we developed. We start the schedulng for three steps n general: (1) Frstly, we employ a predcton technque based on Sngle Exponental Smoothng algorthm to determne when to schedule; (2) Then, we use holstc approach (defne a score model) to determne whch VM should be scheduled; (3) At last, n the most mportant process of schedulng, we use a novel methodology based on vector projecton arthmetc to decde where to allocate the potental VMs. The rest of ths paper s organzed as follows: Secton 2 descrbes related work. Secton 3 detals the algorthms usng n Smart-DRS. Fnally, n Secton 4, we mplement a prototype system to evaluate the performance of Smart-DRS. A summary and plan of our future work are descrbed n Secton 5. 2. RELATED WORKS Dynamc resource schedulng s a prmary problem n vrtual envronment management. It needs dynamcally balances computng capacty across a collecton of hardware resources aggregated nto logcal resource pools. When a VM experences an ncreased load, t should automatcally allocated addtonal resources by redstrbutng VM among the PMs n the resource pool. Ths problem has attracted a lot of attenton from the research communty n the last few years. In [1-3], the authors dscusses respectvely about schedulng scheme makng and executng. They treat VM deployment as a Bn Packng problem and use the Bn Packng dynamc programmng algorthm to solve. Ths method traverses dfferent number of backpacks to fnd the least number of backpacks that can accommodate all tems. However, n fact, VM schedulng problem s more complcated than the Bn Packng problem, because VM has more than two-dmenson factors to care about when place t to the target PM. That means deployng ths method on cloud datacenter has several lmtatons. Mnmze the traffc when schedule the VMs has also been studed. [4] makes use of the network footprnts to generate a good mappng for the data center. The goal s to mnmze the communcaton cost between all the VMs. [5] also ams to mnmze the communcaton traffc n a data center. They consder the topology of a data center and set communcaton cost as the number of swtches between two VMs. Reducng power consumpton s also often consdered. The goal s to reduce the number of actve PMs. There are already some approxmate algorthms proposed to solve ths. The most wdely used s frst-ft [6-8], whch can generate a near-optmal placement. But we thnk that frst-ft has many lmtatons. In a producton data center, VMs are enterng and leavng contnuously and redeployment are requred perodcally. Frst-ft gnores the mgraton cost and may generate unnecessary overhead. We need to take the mgraton cost nto account. Besdes above all, Wood et al. [9] develop an ntegrated VM schedulng system named Sandpper whch s a XEN based automated provsonng system for montorng and detectng hotspots. Thus, t detects when a VM s under-provsoned and ether allots more resources locally or mgrates the VM to a new PM whch s capable of supportng the VM. However, Sandpper s easy to choose a wrong target PM, because Sandpper takes VM mgraton decson based on a metrc, whch t refers to as volume. That means

AN INTEGRATED DYNAMIC RESOURCE SCHEDULING FRAMEWORK 1539 t converts the three dmensonal resource nformaton of PMs nto a sngle dmenson metrc (whch s volume) and then uses ths sngle dmenson metrc for worst ft n a three dmensonal scenaro. In ths process, the nformaton about the shape of the resource utlzaton s lost. Compared to these works mentoned above, our work has the followng man contrbutons: We present an ntegrated schedulng soluton whch contans the major processng procedures of resource management n cloud, such as when to schedule, whch to schedule and where to schedule. We apply SES algorthm to predct workload and score model to choose the mgrated VM. In addton, more mportantly, we employ a novel algorthm based on vector projecton arthmetc to select the potental target PM whch could retan the shape of the mult-dmensonal object representng VM or PM. We develop a prototype system runnng n a small scale cluster wth 32PMs and 3200VMs to evaluate the performance of Smart-DRS. 3. DETAILS OF SMART-DRS In ths secton, we dscuss n detal about our three-step dynamc resource schedulng framework, Smart-DRS. As we know, there are three classc problems [8] to solve n the feld of cloud resource management whch towards smultaneously mnmzng power consumpton and maxmzng QoS performance: Hotspot mtgaton: The goal of hotspot mtgaton s to avod a condton n whch VM runs n a hot PM whch has nadequate resource to meet ts servce level agreements (SLA) requrements. VM mgraton can mtgate ths hotspot, so as to mprove the QoS performance. Load balancng: The goal of load balancng s to avod a stuaton where there s a large dscrepancy n resource utlzaton levels of the PMs. VMs can be mgrated from the hgh workload host to the low workload one, so as to acheve ths balance and mprove the QoS performance. Server consoldaton: The goal of consoldaton s to avod server sprawl many PMs host low-resource-usage VMs. VMs on lghtly loaded host can be packed onto fewer machnes to meet resource requrements, so as to reduce power consumpton. However, for each of the three goals mentoned above, we need to address three mportant questons: when to schedule, whch to schedule and where to schedule. In the followng, we wll dscuss our methods respectvely. 3.1 When to Schedule? As we know, most of tradtonal DRS strateges are based on the threshold whch means the physcal machne surpasses the upper boundary of threshold, then to trgger a

1540 mgraton. Therefore, a VM mgraton could be trggered by nstantaneous peak load, whch leads to the unnecessary waste of mgraton expendture. However, Smart-DRS employs predcton technque based on SES (Sngle Exponental Smoothng) algorthm, whch observes n future load values, f at least k value s above the threshold, we thnk the next load value s above the threshold. SES algorthm s a wdely used forecastng method and an effcent technque that can be appled to tme seres data, ether to produce smooth data for presentaton. SES algorthm s very sutable for our model, because n our system, the tme seres data are a sequence of hstorcal montorng data of resource utlzaton. And the observed phenomenon may be an essentally random process, or t may be an orderly but nosy process. Whereas n the SES the past observatons are weghted equally, exponental smoothng assgns exponentally decreasng weghts over tme. SES removes random perturbatons of tme seres data, then the form of SES s gven by the formula (1), whle the sequence of observatons begns at tme t = 0. y y ( y y ) t1 t t t (1) Where yt 1 s the predcton value at tme t+1, yt s the predcton value at tme t, whle y t s the real value at tme t, s the smoothng factor, and 0 < < 1. Formula (1) can change further to formula (2). y y (1 ) y t1 t t Through the observaton, the above equaton s knd of a recurson equaton, whch can be expanded to formula (3). (2) t1 t y (1 ) y (1 ) y t1 0 t 1 (3) As we can see from formula (3), exponental smoothng forecast value s a weghted sum of all the prevous real observatonal values. That means SES makes use of all the hstorcal data, so t has more stablty and regularty. In addton, we can t avod that there would be some error nformaton n the collected hstorcal montorng data whch wll affect the accuracy of the result. Therefore, we need process the data n advance to delete the nformaton that s not approprate. (4) 1 n 2 1 n n ( y y), y 1 n y 1 y would be arranged n order statstcs. If y satsfes y > k, then y s a bad value, whch should be elmnated, and k s Grubbs crteron coeffcent. Besdes defnng the error nformaton model, we also need to determne the value of and y1 to calculate the predctve value. The Value of : The value of determne the degree of smoothng and how responsve the model s to fluctuaton n the tme seres data. The value of s arbtrary and s de-

AN INTEGRATED DYNAMIC RESOURCE SCHEDULING FRAMEWORK 1541 termned both by the nature of the data and the feelng by the forecaster as to what consttutes a good response rate. A smoothng constant close to zero leads to a stable model whle a constant close to one s hghly reactve. To our knowledge, t s good to set a small value of n order to ncrease the weght of hstorcal data when the tme seres data doesn t have fluctuatons. On the contrary, t s good to set a bg value of to ncrease the weght of recent predcton value when the tme seres data has obvous fluctuatons. The Value of y 1 : In fact, the smaller value of, the more senstve our predcton value wll be on the selecton of ths ntal smoother value y 1. In our method, we defne y 1 to be ntalzed to y 1 when the number of seres data s more than an experental value 15 [10]. Whle the number s less than 15, we defne y 1 to be the average value of the seres data. As shown n formula (5). k y / k, k 15 1 y 1 y, k 15 1 (5) 3.2 Whch to Schedule? Selectng one or more VMs for mgraton s a crucal decson of the resource management. The mgraton process not only makes the VM unavalable for a certan amount of tme but also consumes resources lke CPU, memory and network bandwdth on source and destnaton PMs. Performance of other VMs that are hosted on source and destnaton PMs are also affected due to ncreased resource requrements durng mgraton. Many exstng VM selecton approaches are straghtforward that selectng the canddate VM for mgraton because of resource constraned. The VM whose resource requrements can t be locally fulflled s selected. However, Smart-DRS employs a more holstc approach that CPU utlzaton, memory usage and net bandwdth usage are all consdered before selectng the canddate VM. Reducng the memory data quantty that needs to be mgrated s very mportant. Because that lve mgraton mechansm s a way that teratvely copy VM memory pages to the target host as well as trackng those pages that have been modfed n the copy process. It should ntercept all memory access of the mgratng VM whch wll serously affect the applcaton performance runnng n ths VM. By reducng the amount of data copes n the network can mnmze the total mgraton tme and also mtgate the mpact on applcaton performance. Meanwhle, the VM whose CPU utlzaton rate or net bandwdth consumpton s hgh means t has a great help to mtgate hotspot after mgraton. So Smart-DRS defnes a model to balance these factors: Score(cpu, mem, net) = F(workload(cpu, net), cost(mem)). (6) Where Score(cpu, mem, net) takes the three most mportant factors nto consderaton and t s easy to extend to more dmensons. The score of VM s a functon about workload and cost. Meanwhle, the workload s a functon about CPU and network, and the cost s a functon about memory.

1542 workload cpu net 1 cost mem (7) Where cpu represents the CPU utlzaton rate of VM, net s ts usage of network bandwdth, mem s ts usage of memory. It s clear that we want to choose the VM wth both the hghest workload value and the cost value. For solvng ths problem, we defne: workload cost Score( cpu, mem, net ). workload cost (8) Score s the harmonc mean of workload and cost, and t would be arranged n descendng order, Smart-DRS chooses the one wth hghest score value as the mgrated VM. From formula (8) we can see that ths VM has both the hghest workload and cost whch means t has the most CPU utlzaton rate and network bandwdth but the least memory usage. In other words, the VM of the hghest score value utlze the most physcal resource but easy to mgrate. 3.3 Where to Schedule? After prevous steps, we should choose out the target PM that has enough resources so that t can support the ncomng mgratng VM. To the best of our knowledge, many exstng methods (ncludng heurstc methods) treat ths problem to be smlar to mult-dmensonal Bn Packng problem. However, whle ths method may seem fne on the surface, certan drawbacks and anomales can be uncovered when we analyze deeper. 3.3.1 Mult-dmensonal bn packng algorthm Actually the problem of placng resource-aware VMs among PMs looks smlar to the mult-dmensonal bn packng problem. Exstng lteratures [7, 9-11] all tred to solve VM placement problem wth bn packng algorthm and ts varant. However, they are not exactly the same. Exstng methods consder VM as a three dmensonal object and PM as a three dmensonal box. They try to pack as many objects n the boxes as possble, as well as the number of boxes requred are mnmzed. However, whle packng objects nto a gven box, two objects can be placed sde by sde or one on top of the other. But f we consder VMs as the objects, we can t place VMs lke that. Ths s because once a resource s utlzed by a VM, t can t be reused by any other VM. Fg. 1 shows the man dfference. (a) Intal statement (b) Invald placement (c) Invald placement (d) Reasonable placement Fg. 1. The dfference between VM placement and bn packng problem.

AN INTEGRATED DYNAMIC RESOURCE SCHEDULING FRAMEWORK 1543 For smplcty, we show ths scenaro n 2D model. As we can see form Fg. 1, when VM2 should be placed n ths PM, the only allowed poston s the one shown n Fg. 1 (d). Whle n bn packng algorthm, the postons as shown n Fgs. 1 (b) and (c) are also allowed. Due to ths shortage of bn packng, we employ a novel algorthm to solve the VM placement problem based on Vector Projecton. 3.3.2 Vector projecton algorthm Vector Projecton (VP) model, presented n paper [12], s a novel methodology that can make the process of choosng PMs easer. Based on t, we proposed vector projecton algorthm. In ths algorthm, frstly, we choose three major resources avalable wth the PM, namely CPU, MEM and IO. These resources form the three dmensons of an abstract object. We normalze the resources along each axs. Thus, the total avalable resource can be represented as a unt cube whch s called Normalzed Resource Cube (NRC). Secondly, we should express the resource related nformaton of potental VMs and potental target PMs as a vector wthn the NRC, as shown n Fg. 2. The total capacty of PM s expressed as a vector from the orgn of the cube (0, 0, 0) to pont (1, 1, 1). Ths vector s dentfed as Total Capacty Vector (TCV). Resource Utlzaton Vector (RUV) represents the current utlzaton of resources of a PM The vector dfference between TCV and RUV represents the Remanng Capacty Vector (RCV), whch essentally captures how much capacty s left n the PM. The resource requrement of a VM s represented by Resource Requrement Vector (RRV) whch s the vector addton of normalzed resource requrement vectors of each resource type. In order to measure the degree of mbalance of resource utlzaton of a PM, we defne a Resource Imbalance Vector (RIV) of PM. It s the vector dfference between RUV s projecton on TCV and RUV. RIV of a VM s defned n a smlar way: t s the vector dfference between RRV s projecton on TCV and the RRV. Fg. 2. Depcton of vectors n NRC. Table 1. Vectors name of Aronym. Acronym Name TCV Total Capacty of PM Vector RUV Resource Utlzaton of PM Vector RCV Remanng Capacty of PM Vector RRV Resource Requrement of a VM Vector RIV Resource Imbalance of PM/VM Vector Then, we project NRC on a plane perpendcular to the prncpal dagonal of the cube as shown n Fg. 3. It s easy to see that ths would result n a regular hexagon on the sad projecton plane. After that, we take the projecton of the resource vectors onto the projecton plane to make the resource nformaton of 3D space reduce to 2D plane. We refer to ths as Planar Resource Hexagon (PRH). Corners (0, 0, 0) and (1, 1, 1) of the NRC map to the center of the PRH. Other sx corners maps to the sx vertces of the hexagon as marked n the fgure. Then we take the projecton of tp of the resource vectors TCV, RUV (of PM) and RRV (of VM) onto the projecton plane. The projecton

1544 ponts of these three vectors would be nsde the regular hexagon. For smplcty, the projecton pont of RUV of PM (RRV of VM) wll be referred to as poston of PM (VM) n the PRH. Note that the projecton of tp of TCV s the center of the PRH. By carefully analyss, we can know the regular hexagon on the projecton plane s dvded nto sx regular trangles. Shown n Fg. 4, we named these trangles lke that: CI defnes the regon where the projecton of tp of RUV whose CPU > MEM > IO. Lkewse, MC represents the regon where the projecton of tp of RUV whose MEM > CPU > IO. Other trangles can be dentfed by smlar nequaltes. Further analyss shows that the VMs and PMs whose resource vectors projected n the same trangle or axs have an affnty. Now, we group all of the potental VMs and potental target PMs based on the rules that the RRV of a VM and RCV of a PM are n the same trangle or the same axs and n the opposte drecton. That means they have the characterstcs of complementary resources. Our ultmate goal s to fnd a best complementary target PM for a potental VM n a group, whose RIV s of same magntude as the VM s RIV and s n the opposte drecton. The groups are shown n Table 2. After comparson, we found Fg. 3. Resource vector projecton. Fg. 4. The planar resource hexagon. Table 2. Vector projecton locaton groupngs. Group Name Group Feature 0 CI Imbalanced n CPU 1 IC Imbalanced n IO 2 IM Imbalanced n IO 3 MI Imbalanced n MEM 4 MC Imbalanced n MEM 5 CM Imbalanced n CPU 6 CPU axs Especally mba n CPU 7 IC/CI Imbalanced n CPU & IO 8 IO axs Especally mba n IO 9 MI/IM Imbalanced n MEM & IO 10 MEM axs Especally mba n MEM 11 MC/CM Imbalanced n mem&cpu 12 Orgn P Resource balancng

AN INTEGRATED DYNAMIC RESOURCE SCHEDULING FRAMEWORK 1545 that the trangle s mbalanced n the resource whch the closest resource axs represents. If a trangle has no projecton of tp of vectors, we wll make t wth adjacent trangles nto a new group, because adjacent trangles are wth smlar characterstcs of re-sources. Loop ths process untl every group has resource vectors. 3.3.3 VP algorthm elaboraton Ths segment detals VP algorthm whch s a VM placement algorthm. In secton 3.2, we have ntroduced how to choose a VM to mgrate. And VM placement problem s how to choose a target PM for ths VM. The target PM for ths VM can be chosen based on one of the three goals: hotspot mtgaton, load balancng, and server consoldaton. Algorthm 1: getmgrated_vm (goal) 1. swtch (goal){ 2. case hotspot mtgaton: 3. for all VMs runnng n ths hotspot PM 4. fnd the hghest value of score(x) 5. return VMx 6. case load balancng: 7. order_allpmlst(); /* accordng to the resource utlzaton of every PM n descendng order. */ 8. P=the PM n the top of AllPMlst 9. fnd the hghest value of score(x) n P 10. return VMx 11. case server consoldaton: 12. order_allpmlst(); /* accordng to the resource utlzaton of every PM n ascendng order. */ 13. P=the PM n the top of AllPMlst 14. fnd the hghest value of score(x) n P 15. return VMx 16. } Algorthm 2: gettarget_pm (VM) 1. named the trangle T whch contans VM.RRV 2. for all PMs do 3. f PM.RCV locates n T 4. add ths PM nto PotentalPMlst 5. end f 6. end for 7. f no PM.RCV locates T then 8. T=T+left trangle of T+rght trangle of T 9. end f 10. Remove PMs from PotentalPMlst whch can t support the VM 11. then 12. choose the most complementary PM n PotentalPMlst whose RIV s of the most closest magntude (s the most slghtly less) as the VM s RIV 13. return ths PM Algorthm 1 gets the VM whch should be mgrated away frstly based on three dfferent goals. Then algorthm 1 would pass ths VM as a parameter to algorthm 2. Algorthm 2 frstly generates the potentalpmlst by addng those PMs whose RCV locates n the same trangle wth the VM s RRV. At last, t chooses the most approprate PM n the potentalpmlst whose RIV s of same magntude as the VM s RIV and s n the opposte drecton. 4. EXPERIMENTAL RESULTS For evaluatng the performance of Smart-DRS, we have mplemented a prototype system as dscussed n Secton 3. Besdes, we buld a small-scale cluster whose detal nformaton s lsted as follows:

1546 Table 3. Experments envronment. PM VM CPU VMM Memory Network Numbers Numbers Frequency Verson 32 3200 3.3GHz 4GB 100Mb/s Xen 3.0.3 The major goal of our desgn s to evaluate the performance of Smart-DRS strategy whle appled n data center envronment. In order to avod tme-consumng and complcated mplementaton n code wthout knowng potental effects of the modfcaton, we choose some mature open source projects n our system, such as Gangla n montor module and Xen Moton n executon module. The detaled system archtecture s descrbed n Fg. 5. Fg. 5. The archtecture of experment prototype system. 4.1 Evaluaton of When to Schedule In order to evaluate the performance of predctng algorthm, we need some of convctve host load samples. Fortunately, Dnda and O Halloran from Carnege Mellon Unversty offered us plenty of load samples by long-term tracng wth many knds of machnes n a cluster system. We choose the day s collecton of load tme seres on August 18, 2010 as our test samples. In the frst experment, as shown n Fg. 6 (a), we contnuously montor 32 PMs for 21 mnutes and record the real load value. After that, we respectvely calculate the forecastng value for average when = 0.1, 0.3, 0.5, the Mean Relatve Error (MRE) s correspondngly 13.6%, 9.7% and 7.3%. That means SES algorthm has an acceptable predctng accuracy whle appled n a small-scale datacenter and = 0.5 s a better stuaton n our experment envronment. In the second experment, as shown n Fg. 6 (b), we compare Smart-DRS wth other classc forecastng algorthms such as AR (Autoregressve), MA (Movng Average), ARMA (Autoregressve Movng Average), ARIMA (Autoregressve Integrated Movng Average) and BP (Back Propagaton) neural network. We can see that ARMA predcton algorthm has the least mean relatve error, whch means t has the most accuracy, be-

AN INTEGRATED DYNAMIC RESOURCE SCHEDULING FRAMEWORK 1547 (a) Compared wth tself of predcton effcency. (b) Compared wth other predcton methods. Fg. 6. Evaluaton of when to schedule performance. cause t combnes AR s advantage wth MA s. Then our Smart-DRS follows, t has more accuracy that BP neural network whch s a knd of contnuous practcng algorthm. Even Smart-DRS s a relatvely smple model, t has a relatvely good performance, and the mean relatve error of t s acceptable. 4.2 Evaluaton of Whch to Schedule SandPper s a Xen based automated provsonng system [6]. It takes VM mgraton decson based on a metrc, whch refers to as sand_volume. 1 1 1 sand _ volume 1cpu 1mem 1net (9) Where cpu, mem and net are the correspondng normalzed utlzatons of the resources (n ths paper, for smplcty, we just consder PM as a 3D object). Accordng to our experence, n many experments, authors would lke choose the VM whose sand_ volume s the hghest as the mgrated VM. Because sand_volume value represents the resource utlzaton of the VM and mgratng the hghest one helps a lot to mtgate the resource utlzaton rate. However, dfferent VM choosng strategy wll affect the mgraton tme and mgraton numbers. In ths experment, we carry out a seres of 20 groups tests ncludng dfferent ntal statements of our experment cluster, such as hotspot, load unbalancng, low resource utlzaton and so on. We count the mgraton tme and mgraton numbers from ntal state to the fnal state between SandPper model and our score model of Smart-DRS. Fg. 7 (a) tells us that Smart-DRS can save much more mgraton tme whle just a lttle hgh n mgraton numbers compared to SandPper shown n Fg. 7 (b). Because Smart-DRS would lke to choose the VM whch utlzes hgh cpu and net bandwdth but low memory resource. Smart-DRS can reduce much more mgraton tme at the sacrfce of a lttle more mgraton numbers.

1548 (a) Mgraton Tme experment. (b) Mgraton Number experment. Fg. 7. Evaluaton of whch to schedule performance. 4.3 Evaluaton of Where to Schedule In ths experment, we try to evaluate the performance of VM placement. Frst of all, we should clam that we have defned Balance to represent the degree of load balancng of each machne and SYS balance to represent the degree of load balancng of the whole system (a hgher value s better). Balance (10) 2 2 2 [ ]/2 cpu mem o n cpu cpu / n cpu 1 n mem mem / n mem 1 n o o / n o 1 SYS (11) n balance Balance n (12) 1 Ths experment compares the performance of Smart-DRS wth two classcal Bn Packng algorthms used n dynamc resource schedulng, Best Ft Decreasng (BFD) and Frst Ft Decreasng (FFD). We measure the algorthm executon tme and SYS balance of each algorthm. As shown n Fg. 8, the BFD and FFD consume much more executon Fg. 8. Performance comparsons among varous schedulng algorthms.

AN INTEGRATED DYNAMIC RESOURCE SCHEDULING FRAMEWORK 1549 tme and acheve no better performance n SYS balance than Smart-DRS. The reason s that they have to sort all of the potental VMs and target PMs frstly accordng to ther resource utlzaton. 4.4 Overall Evaluaton In the last experment, we assess the performance of Smart-DRS n overall stuaton. By contnuously montorng the resource utlzaton of cpu, mem, net of 32 PMs, we could observe the dfference between before schedulng and after schedulng. In order to carry out ths experment, we have set 0.8 as the upper threshold and 0.2 as the lower threshold. (a) Before schedulng (load balancng). (b) After schedulng (load balancng). (c) Before schedulng (server consoldaton). (d) After schedulng (server consoldaton). Fg. 9. The overall evaluaton between before and after schedulng. Fg. 9 shows that our schedulng algorthm really works. It s clear that the load of system s mbalanced and the resource utlzaton of several machnes exceeds 0.8 before schedulng as shown n Fg. 9 (a). Whle n Fg. 9 (b), after schedulng, the load of system s more balanced and none of resource utlzaton exceeds the upper threshold. Smlarly, n Fg. 9 (c) we can see that the resource utlzaton of several machnes s under 0.2 whle n Fg. 9 (d), these low utlzaton machnes are taken offlne and the others are load balancng. 5. CONCLUSION In ths paper, we studed the resource management method n on-demand cloud envronment and we have presented an ntegrated dynamc resource schedulng framework named Smart-DRS. It employs SES algorthm to predct the resource utlzaton of PMs

1550 n order to avod tny and temporary load peak value trggerng unnecessary mgraton. The experment result shows that predcton value s pretty close wth the real value. Then Smart-DRS creates score model to determne whch VM to choose. And the result tells us that t can save much more mgraton tme whch s very mportant to mprove QoS performance. At last, Smart-DRS employs VP algorthm to make the mgraton scheme. The reason why we choose VP algorthm s that t s a novel theory whch can be used to make the process of choosng PMs easer and more approprate. The experment result tells us that t works really well and t s a knd of low cost and effcent method. Nevertheless, our framework has some lmtatons that we plan to address n the future: (1) Frstly, we can t employ Smart-DRS on a large scale cloud data center, so we don t know ts scalablty; (2) Then, we wll comment on the computaton and space complexty of vector projecton algorthm n future work; (3) At last, we don t talk about the user prorty. In the real world, t s very lkely that some hgh prorty users or users who pay more for the resources need to occupy the whole physcal machnes. We can t move other VMs to these machnes even f they are actually dle. In addton, varous other measurements and optmzaton strateges wll need to be explored n the future. REFERENCES 1. F. Hermener, X. Lorca, J. M. Menaud, G. Muller, and J. Lawall, Entropy: a consoldaton manager for clusters, n Proceedngs of ACM SIGPLAN Internatonal Conference on Vrtual Executon Envronments, 2009, pp. 41-50. 2. C. Hyser, B. Mckee, R. Gardner, and B. J. Watson, Autonomc vrtual machne placement n the data center, Hewlett Packard Laboratores, 2007, pp. 189-195. 3. R. Nathuj and K. Schwan, VrtualPower: Coordnated power management n vrtualzed enterprse systems, n Proceedngs of the 21st ACM SIGOPS Symposum on Operatng Systems Prncples, 2007, pp. 265-278. 4. J. D. Sonnek, J. B. S. G. Greensky, R. Reutman, and A. Chandra, Starlng: Mnmzng communcaton overhead n vrtualzed computng platforms usng decentralzed affnty-aware mgraton, n Proceedngs of Internatonal Conference on Parallel Processng, 2010, pp. 228-237. 5. X. Meng, V. Pappas, and L. Zhang, Improvng the scalablty of data center networks wth traffc-aware vrtual machne placement, n Proceedngs of Internatonal Conference on Computer Communcatons, 2010, pp. 1154-1162. 6. T. Wood, P. J. Shenoy, A. Venkataraman, and M. S. Yousf, Sandpper: Blackbox and gray-box resource management for vrtual machnes, Computer Networks, Vol. 53, 2009, pp. 2923-2938. 7. M. Cardosa, M. Korupolu, and A. Sngh, Shares and utltes based power consoldaton n vrtualzed server envronments, n Proceedngs of IFIP/IEEE Internatonal Symposum on Integrated Network Management, 2009, pp. 327-334. 8. A. Verma, P. Ahuja, and A. Neog, Power-aware dynamc placement of hpc applcatons, n Proceedngs of the 22nd Annual Internatonal Conference on Super- Computng, 2008, pp. 175-184. 9. T. Wood, P. J. Shenoy, A. Venkataraman, and M. S. Yousf, Black-box and gray-

AN INTEGRATED DYNAMIC RESOURCE SCHEDULING FRAMEWORK 1551 box strateges for vrtual machne mgraton, n Proceedngs of the 4th ACM/USE- NIX Symposum on Networked Systems Desgn and Implementaton, 2007, pp. 229-242. 10. M. Mshra, A. Das, P. Kulkarn, and A. Sahoo, Dynamc resource management usng vrtual machne mgratons, Communcatons Magazne, Vol. 50, 2012, pp. 34-40. 11. F. Hermener, S. Demassey, and X. Lorca, Bn repackng schedulng n vrtualzed datacenters, n Proceedngs of the 17th Internatonal Conference on Prncples and Practce of Constrant Programmng, 2011, pp. 27-41. 12. N. Bobroff, A. Kochut, and K. Beaty, Dynamc placement of vrtual machnes for managng sla volatons, n Proceedngs of the 10th Internatonal Symposum on Integrated Network Management, 2007, pp. 119-128. 13. M. Wang, X. Meng, and L. Zhang, Consoldatng vrtual machnes wth dynamc bandwdth demand n data centers, n Proceedngs of IEEE Internatonal Conference on Computer Communcatons, 2011, pp. 71-75. 14. M. Mshra and A. Sahoo, On theory of VM placement: Anomales n exstng methodoges and ther mgraton usng a novel vector based approach, n Proceedngs of IEEE 4th Internatonal Conference on Cloud Computng, 2011, pp. 275-282. 15. J. L, J. Peng, Z. Le, and W. Zhang, An energy-effcent schedulng approach based on prvate clouds, Journal of Informaton Computatonal Scence, Vol. 8, 2011, pp. 716-724. 16. L. Xu, W. Chen, Z. Wang, and S. Yang, Smart-DRS: A strategy of dynamc resource schedulng n cloud data center, n Proceedngs of IEEE Internatonal Conference on Cluster Computng Workshops, 2012, pp. 120-127. 17. T. Imada, M. Sato, and H. Kmura, Power and qos performance characterstcs of vrtualzed servers, n Proceedngs of the 10th Internatonal Conference on Grd Computng, 2009, pp. 232-240. Le Xu () s now a Ph.D. student n the College of Computer Scence and Technology, Zhejang Unversty. Hs current research nterests nclude operatng system, vrtualzaton and cloud nfrastructure. Zonghu Wang ( 总 辉 ), born n March 1979, s a Lecturer n the College of Computer Scence and Engneerng at Zhejang Unversty n Hangzhou, Chna. He receved hs Ph.D. n the College of Computer Scence and Technology at Zhejang Unversty n 2007. Hs research nterests focuse on cloud computng, dstrbuted system and computer archtecture.

1552 Wenzh Chen ( 陈 ) born n 1969, receved hs MS and Ph.D. degrees from Zhejang Unversty, Hangzhou, Chna. He s now a Professor and Ph.D. supervsor of College of Computer Scence and Technology of Zhejang Unversty. Hs areas of research nclude computer archtecture, system software, embedded system and securty.