Dynamic Pricing and Profit Maximization for the Cloud with Geo-distributed Data Centers

Transcription

1 Dynamic Picing an Pofit Maximization fo the Clou with Geo-istibute Data Centes Jian Zhao, Hongxing Li, Chuan Wu, Zongpeng Li, Zhizhong Zhang, Fancis C.M. Lau The Univesity of Hong Kong, Univesity of Calgay, Abstact Clou povies often choose to opeate atacentes ove a lage geogaphic span, in oe that uses may be seve by esouces in thei poximity. Due to time an spatial ivesities in utility pices an opeational costs, iffeent atacentes typically have ispaate chages fo the same sevices. Clou uses ae fee to choose the atacentes to un thei jobs, base on a joint consieation of monetay chages an quality of sevice. A funamental poblem with significant economic implications is how the clou shoul pice its atacente esouces at iffeent locations, such that its oveall pofit is maximize. The challenge escalates when ynamic esouce picing is allowe an long-tem pofit maximization is pusue. We esign an efficient online algoithm fo ynamic picing of VM esouces acoss atacentes in a geo-istibute clou, togethe with job scheuling an seve povisioning in each atacente, to maximize the pofit of the clou povie ove a long un. Theoetical analysis shows that ou algoithm can scheule jobs within thei espective ealines, while achieving a time-aveage oveall pofit closely appoaching the offline maximum, which is compute by assuming that pefect infomation on futue job aivals ae feely available. Empiical stuies futhe veify the efficacy of ou online pofit maximizing algoithm. I. INTRODUCTION Recent yeas have witnesse the polifeation of clou computing platfoms, sevices an applications [] [2] [3]. To bette seve the computing emans fom uses in iffeent geogaphical egions, it is common fo a clou povie to host multiple atacentes in a numbe of selecte locations. Given the iffeent opeational costs acoss sevice egions, esouces (e.g., vitual machines) ae natually pice iffeently acoss ata centes [2]. Uses of the clou system can stategically ecie the atacentes to un thei jobs in, base on the esouce pices an the esie quality of sevice (e.g., communication elays between the use s location an the atacentes). How the clou povie shoul pice its esouces in atacentes istibute acoss iffeent locations such that the oveall pofit is maximize is a poblem of funamental impotance. As compae to fixe pices (e.g., Amazon on-eman instances), ynamic picing that eflects the ealtime supplyeman elationship (e.g., Amazon spot instances) epesents a moe pomising chage stategy that can bette exploit use payment potentials an thus lage pofit gains at the clou povie. Une the objective to maximize the oveall pofit in The eseach was suppote in pat by gants fom Hong Kong RGC une the contacts HKU 7782 an HKU 7853, an in pat by the Natual Sciences an Engineeing Reseach Council of Canaa (NSERC) /4/$ IEEE. the clou, it is howeve non-tivial to ecie a ynamic pice fo VMs in each atacente at a given time, which is intimately connecte to ecisions on seve ight-sizing (tuning seves on/off) an job scheuling among iffeent atacentes. The challenge escalates when we want to pusue timeaveage pofit maximization ove a long un of the system, with ynamically aiving use jobs with heteogeneous execution times, an base on online ecision making. A numbe of intiguing questions ae involve: What is the stategy fo each use to select the clou atacente fo its job execution, in oe to maximize its own utility? Given the use stategy, how shoul the clou ynamically pice its VMs an ecie the numbe of active seves in each atacente at any time such that the jobs ae maximally seve an its pofit is maximize ove time? In this wok, we answe these questions by jointly moelling job scheuling, VM picing an seve povisioning ecisions as an integate stochastic optimization famewok base on Lyapunov optimization theoy [4]. An efficient online algoithm is esigne to guie the opeational ecisions of the clou povie to pusue maximal time-aveage pofit ove the long un. Base on igoous theoetical analysis, we emonstate that the algoithm has the following esie popeties: () The algoithm guaantees no job opping une two mil conitions as pesente in Sec. IV-B, while all the accepte jobs can be complete within thei espective completion ealines; (2) the algoithm achieves a time-aveage oveall pofit fo the clou povie, which can appoach the offline maximum abitaily closely. Note that the latte is compute une the stong assumption that complete infomation of all job aivals, incluing those in the futue, ae magically available. To ou knowlege, this wok is among the fist to esign efficient stategies fo joint ynamic picing, job scheuling, an esouce povisioning in the clou computing liteatue, an among the fist to hanle jobs with vaiable lengths une the Lyapunov optimization famewok. In paticula, we consie a clou with vaious VM configuations, whose opeational costs vay in both the tempoal an spatial omains. We aess ynamic aivals of jobs into the clou, with vaious equiements on types an lengths of occupation of iffeent VMs, as well as iffeent job completion ealines. A salient contibution in ou Lyapunov optimization appoach is that, we allow the execution time of each job to be longe than the inteval of online ecision making, such that ecisions

2 2 in consecutive ecision intevals ae stongly coelate, beyon what the stana Lyapunov optimization famewok can hanle. Employing a new esign of the ynamic algoithm in two time scales, we can still ensue its close-to-optimal pefomance, base on igoous theoetical analysis. It is notewothy that ou algoithm has funamental iffeence fom the ingenious wok [5] with two-time-scale scheuling, in that we nee no expectation into the futue to be extacte fom histoical ata. Instea, ou famewok makes ynamic ecisions just base on the cuent status of the system. The emaine of the pape is oganize as follows. Sec. II pesents the system moel. The online algoithm is esigne in Sec. III. The pefomance of the algoithm is analyze in Sec. IV. Sec. V pesents the simulation esults. We eview elate liteatue in Sec. VI an conclue the pape in Sec. VII. II. MODEL &NOTATION A. The Clou System Moel Consie a clou povie with a set D (with size D = D ) of geo-istibute atacentes, inexe by whee D. Each atacente has N homogeneous seves. The system opeates in a time-slotte fashion, fo t =0,,...,T. A set H (with size H = H ) of istinct types of vitual machine (VM) instances ae povie in the clou, each with a specific set of configuations of CPU, memoy, an stoage, chaacteizing heteogenous VM instance povisioning in the eal wol such as in Amazon EC2 [2]. Each seve in a atacente hosts VMs of the same type in a time slot, which can change acoss iffeent time slots [6]. Let n h enote the maximum numbe of type-h VMs that a seve of atacente can simultaneously host. Datacentes eceive VM equests fom customes in the fom of jobs. Each job R is a pai (h,w ), whee h His the type of VM equeste; w [w min,w max ] is the numbe of time slots equeste an is efee to the wokloa of the job. The set of possible job types is R with R = R. As a Sevice Level Ageement (SLA), the clou povie guaantees that the maximum job scheuling latency is boune by l, i.e., the elay fom the time the job is submitte to a atacente to the time it is allocate a VM, will not excee l. Clou customes esie in a set of geo-istibute zones J with size J = J. The utility obtaine by customes in zone j when type- jobs ae seve by atacente is U j, ( ),j J, D, R, which is a iffeentiable, concave utility function. B. The Clou Povie s Solution Space We aim to esign ynamic, optimal algoithms fo the clou povie to stategically make the following opeational ecisions in each atacente at each time slot: (i) Font-en job picing: What pices shoul be chage to each type of jobs with a specific wokloa? (ii) Job scheuling: How many jobs of each type shoul be scheule fo execution in each atacente? (iii) Seve/VM povisioning: How many seves shoul be tune on, an what type of VMs shoul each active seve povision? The goal is oveall pofit optimization fom all atacentes ove the long un. Job Picing. Let p (t) [0,p,max ] be the pice chage to a type- job at atacente at time t, uppe-boune by p,max, which will be elate to customes maximum value fo a type- job at atacente. Given the job pice p (t), customes in zone j will equest (t) type- jobs fom atacente, fo maximizing thei suplus (total utility minus total chages) une the chaging pices p (t): (t) =agmax j, a [U j, ( (t)) p (t) (t)]. () D The total numbe of type- jobs atacente eceives is j J aj, (t), with total wokloa w j J aj, (t). The job aival ate (t) is uppe-boune by a max. Job Scheuling. Each atacente maintains R queues of unscheule wokloa, Q, each coesponing to a istinct job type, R. Upon aival of a type- job at atacente, w units of wokloa ae appene to Q ; when a job is scheule fo execution, a unit wokloa epats fom Q in that time slot. Let μ (t) enote the numbe of type- jobs scheule to un in atacente in time slot t. Once scheule, the job will occupy a VM of type h fo w consecutive time slot(s). Let μ (t ) enote the numbe of type- jobs scheule befoe t, which ae still unning on atacente in t. We moel the potential opping of a job when its SLA equiement l (max scheuling elay) cannot be met. Let G (t) enote the numbe of unscheule jobs of type in atacente, which ae oppe in t, 0 G (t) G max, whee G max is the maximum numbe of jobs allowe to op in one time slot, In pactice, a clou may neve op a use s job. The op in ou moel can be unestoo as follows: The clou maintains a set of egula esouces ( D N VMs) while keeping a set of backup esouces, whose povisioning can be expensive. When a job is oppe ue to not being scheule using the egula esouces when its esponse elay is ue, the clou uses its expensive backup esouces to seve the job, subject to a cost η ( the job op penalty ) to seve one type- job. The SLA equiement can be fomulate as follows: Each type- job is eithe scheule o oppe (subject to a penalty) befoe its maximum scheuling elay l. (2) Let Q (t) be the total unpocesse wokloa of type- jobs in atacente at t. It is upate ove time as follows: Q (t +)=max{q (t) μ (t) μ (t ) w G (t), 0} + w (t). (3) j J Hee, μ (t) is the total numbe of type- jobs newly scheule to un on VMs of type h at the beginning of time slot t; fo each of these jobs, one unit of wokloa is euce fom Q (t) afte the job has been unning fo the time slot. μ (t ) is the total numbe of left-ove type- jobs still unning in atacente ; fo each of these jobs, one unit of wokloa is also euce fom Q (t) afte it has been unning fo that time slot. The

3 3 TABLE I IMPORTANT NOTATIONS D set of atacentes H set of VM types R set of all job types N # of seves at D h VM type fo type- jobs w size of a type- job l Max job scheuling elay J set of custome zones n h Max # of type h VMs a seve in atacente can host p (t) pice chage to a type- job at t in (t) # of amitte type jobs fom zone j at t in μ (t) # of new type jobs execute at atacente at time slot t μ (t ) # of type jobs at atacente left ove fom ealie G (t) # of oppe jobs in Q at time slot t Nh (t) # of active seves fo type-h VMs in at time t c (t) unit powe cost fo unning one seve at atacente in t η penalty fo opping a type- job P (t) clou s pofit at time t Q queue of unscheule type wokloa at atacente Z vitual queue fo bouning queueing elay in Q ɛ p,max a max μ,max G max peset constant fo contolling queueing elay in Q max pice fo type- jobs in atacente max # of type- jobs aiving in one slot max # of type- jobs allowe to be scheule in one slot max # of type- jobs allowe to be oppe in one slot op of G (t) unscheule jobs bings a euction of w G (t) units of wokloa fom Q (t). μ (t ), R, D, ae known in t, base on hitheto scheuling ecisions an the wokloa size of each scheule job. μ (t), G (t), an p (t) (which ecies (t)), R, D, ae ecision vaiables ou algoithm juiciously computes in each time slot, not only to maximize the pofit, but also to guaantee that the scheuling elay of each job of type is within its ealine l. In paticula, if the maximum queueing elay of each unit of wokloa in Q can be boune by l, then the maximum scheuling elay fo each incoming type- job is also boune within l. Seve/VM Povisioning. Let Nh (t) enote the numbe of active seves in atacente configue to povision VMs of type h in time slot t. These seves can be use to seve jobs of type-, whee h = h. Wehave Nh(t) (μ (t)+μ (t ))/n h. :h =h We ae inteeste in the minimum numbe of seves equie to meet the VM emans, assuming an efficient inta-atacente VM migation algoithm [7]) that helps move unning VMs fom one seve to anothe, fo eucing the numbe of active seves. C. The Pofit Maximization Poblem The clou povie s net pofit is the iffeence between the evenue an the costs. The total evenue by taking in jobs of iffeent types in t is D R j J aj, (t)p (t). We consie powe consumption in opeating seves as the majo component of opeational costs in a atacente [8]. Let c (t) be the unit cost of opeating one seve in atacente in time slot t, which is natually time vaying an location epenent. The total cost in the clou in t is D c (t) N h(t). A penalty of η is enfoce fo opping a job of type-, with η p,max, R, D. Hence, expenitue on penalty occus in time slot t if thee ae oppe jobs, with the total amount of D R η G (t). The net pofit of the clou povie in time slot t is: P (t) = (t)p (t) c (t) Nh(t) D R j J D η G (t). D R The time-aveage expecte pofit of the clou is: P (t) lim sup T E [P (t)]. T T t=0 The pofit maximization pusue by the clou is theefoe: max : P (t) (4) s.t. : 0 p (t) p,max, R, D,t [,T]; (5) 0 G (t) G max, R, D,t [,T]; (6) Nh(t) N, D,t [,T]; (7) (μ (t)+μ (t ))/n h Nh(t), :h =h h H, D,t [,T]; (8) μ (t) 0, R, D,t [,T]; (9) Nh(t) Z + 0, h H, D,t [,T]; (0) T lim T T t=0 E{w j J [p (t)]} < T lim E{μ (t)+μ (t )+w G (t)}, T T t=0 R, D; () Constaint (2). This optimization poblem is fo the clou povie to choose an appopiate pice fo each type of jobs at each atacente (p (t)), the best numbe of seves to povision each type of VMs in each atacente (Nh (t)), the optimal numbes of jobs of each type to scheule an to op (μ (t) an G (t)), in each t at each atacente, to maximize its time-aveage pofit. Constaint (7) ensues that the total numbe of active seves in each atacente is boune by the numbe of onpemise seves. Constaint (8) specifies that the total numbe of newly scheule an left-ove jobs in a atacente, each equiing a type-h VM, oes not excee the numbe of type-h VMs povisione. Constaint () guaantees the stability of job queue Q, by ensuing that the aveage aival ate is no highe than the aveage epatue ate [4]. Table I summaizes the notations fo ease of efeence. III. THE DYNAMIC PROFIT MAXIMIZATION ALGORITHM We now esign an online algoithm to solve the pofit maximization poblem in (4). A. Aessing SLA Requiements To guaantee that the wost-case queueing elay in each wokloa queue Q, R, D, is boune by l, we associate each wokloa queue Q with a vitual queue Z (t), base on the ɛ-pesistent sevice queue technique fo elay

4 4 bouning [9]. When the queue backlogs of Q an Z, R, D, ae boune, the jobs queueing elays ae boune. The backlog of the vitual queue is initially Z (0) = 0, an then upate as follows: Z (t +)=max[z (t)+ Q (t)>0(ɛ μ (t) μ (t )) w G (t) Q (t)=0μ,max, 0]. (2) Hee the inicato function Q (t)>0 is when Q (t) > 0, an 0 othewise. Similaly, Q (t)=0 is when Q (t) =0, an 0 othewise. ɛ is a pe-efine constant that is no lage than w a max an can be gauge to contol the queueing elay boun. μ,max is the maximum numbe of type- jobs that can un simultaneously in atacente, with μ (t) +μ (t ) μ,max. By esigning a ynamic algoithm that guaantees the lengths of queues Z an Q ae boune ove time, we ae able to guaantee the queueing elay of wokloa queue Q is boune by l. The ationale can be intuitively explaine as follows: Let Z,max, Q,max be the boun of queues Z, Q, espectively. Consie wokloas aiving at any time slot t. In the subsequent l time slots afte t, ifq eceases to 0, the wokloas ae seve within l time slots; othewise, Z has a constant aival ate ɛ, an the same epatue ate μ (t)+μ (t )+w G (t) as that in the wokloa queue Q. Fo the inteval of l time slots following t, the total aivals into queue Z minus the total epatues is smalle than o equal, i.e., ɛ l t+l τ=t+ [μ (t)+ μ (t) +w G (t)] Z,max. At any time slot t, as the positions of wokloas aiving at time slot t in queue Q woul not excee the boun Q,max, when the total epatue numbe uing the l time slots following t is at least Q,max, i.e., t+l t+ [μ (t)+μ (t )+w G (t)] Q,max, jobs aiving at t will be seve within these l time slots. Hence when ɛ l Z,max Q,max (i.e., l = (Z,max + Q,max )/ɛ ), which guaantees t+l t+ [μ (t)+μ (t )+w G (t)] ɛ l Z,max Q,max, all jobs ae scheule with elays of at most l time slots. to the queue length boun Z,max B. Dynamic Algoithm Design In an online algoithm, we compute instantaneous values of the ecision vaiables, while seeking to solve the optimization in (4) that involves time-aveage vaiable values. To satisfy constaint (), we nee to guaantee that each wokloa queue Q is stable ove time [0]. To maximize the time-aveage objective function base on ecisions in each time slot, we esot to the ift-plus-penalty famewok in Lyapunov optimization [4], a classic technique fo tanslating a long-tem time-aveage optimization poblem into a seies of simila one-shot optimization poblems. In paticula, let Θ(t) =[Q(t), Z(t)] be the vecto of all queues in the system, whee Q(t) an Z(t) ae the vectos of wokloa queues Q (t) an vitual queues Z (t), espectively, R, D.We efine a Lyapunov function as follows: L(Θ(t)) = 2 [ (Q (t) 2 + Z (t) 2 )]. R D The one-slot conitional Lyapunov ift is Δ(Θ(t)) = E{L(Θ(t +)) L(Θ(t)) Θ(t)}. Following the ift-plus-penalty famewok in Lyapunov optimization [4], we minimize an uppe boun fo the following expession in each time slot t, with the obsevation of the queue states ([Q(t), Z(t)]), the numbe of jobs still unning in atacentes (μ (t ), R, R), an costs of unning seves in the atacentes (c (t), D), such that a lowe boun fo P (t) is maximize (see Chapte 5 in [4]): Δ(Θ(t)) VP(t). Hee, V is a non-negative paamete chosen by the clou to contol the taeoff between the pofit an the SLA guaantee. A lage V leas to a highe time-aveage pofit but a highe queueing elay at the same time. Squaing the queueing laws (3) an (2), we can eive the following inequality (etaile steps in technical epot []): Δ(Θ(t)) VP(t) B + (t)[w Q (t) Vp (t)] D R j J + V c (t) Nh(t) D D,R[μ (t)+μ (t )][Q (t)+z (t)] + [Vη w Q (t) w Z (t)]g (t) (3) D R whee B = 2 [(w a max ) 2 +2(μ,max + w G max ) 2 +(ɛ ) 2 ] R D is a constant. Ou algoithm seeks to minimize the RHS of inequality (3), to minimize the uppe boun fo Δ(Θ(t)) VP(t), an thus to maximize the lowe boun of P (t). The boun of wokloa queues Q s an vitual queues Z s can also be guaantee in this pocess (Sec. IV), such that constaint () an the SLA equiements of each type of jobs ae satisfie. In paticula, in each time slot t, the algoithm obseves the queues Q (t) an Z (t), the cuent unit costs of unning seves in atacentes c (t), the numbe of active type- jobs at atacente μ (t ), an ecies the optimal values of p (t),μ (t), G (t) an Nh (t), by solving the following oneshot optimization poblem: min: RHS of (3) (4) s.t.: Constaints (5)(6)(7)(8)(9)(0). A iffeence between this wok an pevious wok using Lyapunov optimization is that the pevious wok usually assume each job can be complete in one time slot, while we moel the moe geneal scenaio in which a job may take moe than one time slot to finish (an they can not be pematuely teminate once scheule to un on the equie VMs). Peviously scheule jobs may still be unning in atacentes an occupying VMs. This constains the contol ecisions in the cuent time slot. We pesent the etaile contol ecisions in the following.

5 5 A caeful investigation of the RHS of (3) eveals that optimization (4) can be equivalently ecouple into thee types of inepenent optimization (excluing constant tems), ealing with (a) font-en job picing, (b) job opping, an (c) job scheuling an seve/vm povisioning, espectively. (a) Font-en Picing: It ecies the pice chage to a type job in each atacente. To minimize the RHS of (3), the pat elate to pices is as follows: min (t)[w Q (t) Vp (t)]. (5) D R j J Recall that the numbe of type- jobs uses in a zone j submit to each atacente, (t), is ecie by the pices p (t) s in iffeent atacentes, in oe to maximize thei suplus, as given in (). The maginal suplus is eive as (U j, ) ( (t)) p (t). As U j, ( ) is a iffeentiable concave function, (U j, ) ( (t)) is non-inceasing. When p (t) > (U j, ) (0), customes in zone j will not un thei type jobs in atacente, i.e., =0. When p (t) (U j, ) (0), the numbe of type- jobs that customes in zone j will sen to atacente is compute by setting the maginal suplus to zeo, as (t) = (U j, ) (p (t)), whee (U j, ) ( ) enotes the invese function of (U j, ) ( ). We eplace (t) in (5) by max{0, (U j, ) (p (t))}, an optimization (5) is now on the pice vaiables p (t) s only. Let ˆp j, enote (U j, ) (0), i.e., the pice value une which uses in zone j will not equest type- jobs fom atacente. In geneal, the J pice values ˆp j,, j J can be sequence fom the lowest one to the highest one, ˆp j, ˆp j 2, [ˆp jm,... ˆp j J,. Fo pices among egion, ˆp j m+, ], m J, uses in zones fom j m+ will equest VMs fom atacente, an the to zone j J coesponing optimization poblem is as follows: min s.t. J (U j i, i=m+ ˆp jm, Fo each egion [ˆp j m, ) (p (t))[w Q (t) Vp (t)] p (t) ˆp j m+,. (6), ˆp jm+, ], m J, thee is an optimization poblem. Thee ae in total J optimization poblems. Among iffeent pice egions, the objective function changes ue to the eason that uses in some zones may not use the sevice. The optimal picing stategy in the esulte J solutions is the one achieving the minimum objective function value. (b) Job Dopping: The numbe of jobs oppe fom queue Q in t, G (t), R, is eive by solving the following minimization poblem: min: [Vη w Q (t) w Z (t)]g (t) (7) s.t.: Constaint (6). The optimal solution to the above LP is: { G (t) = G max, if Q (t)+z (t) > Vη w ; 0, if Q (t)+z (t) Vη (8) w. The above stategy inicates that a type- job is less likely to be oppe in t when the penalty of opping a type- job, η, is lage, an jobs equiing smalle unning times, w, ae less likely to be oppe too. In Theoem 2 to be pove in Sec. IV, we will show that ou scheuling algoithm guaantees zeo job opping, i.e., all jobs amitte into the clou ae successfully pocesse in time, une two conitions: () At any atacente, the accumulate wokloa of any type of jobs since the last time slot when wokloa fom the espective queue is scheule, can all be ispatche to un on seves the next time when the queue is being scheule; (2) the op penalty is high enough to make the clou moe willing to tun on seves than to op jobs, even though the powe cost eaches the maximum value. (c) Job Scheuling an Seve/VM Povisioning: Decisions on μ (t) an Nh (t) in atacente ae mae by solving the following minimization poblem: min: V c (t)nh(t) R[Q (t)+z (t)]μ (t) (9) s.t.: Constaints (7)(8)(9)(0). (9) is a joint job scheuling an seve/vm povisioning poblem. It can be solve by fist conveting to a pue seve/vm povisioning poblem an then eciing job scheuling base on the seve/vm povisioning ecisions. Jobs of iffeent types scheule to atacente, whee h = h, compete fo type-h VMs povisione in the atacente, as given in constaint (8). Suppose the numbe of seves configue to povision type-h VMs in atacente, N h (t), is known. To minimize (9), we shoul maximally scheule jobs of type h, whose obseve value of Q (t)+z (t) is the lagest among all types of jobs equiing type-h VMs, onto the povisione type-h VMs, i.e., h = agmax :h =h [Q (t)+z (t)], (20) whee ties ae boken anomly. The numbe of type-h jobs we can scheule in t is ecie by constaint (8), at μ h (t) =n hnh(t) μ (t ), h H. (2) :h =h That is, except VMs occupie by left-ove jobs, all othe typeh VMs shoul be use to seve type-h jobs, an no othe types of jobs ae scheule, i.e., μ (t) =0, h, h H. (22) Hence, the secon pat of (9) can be expesse using vaiables Nh (t) s: R[Q (t)+z (t)]μ (t) = [Q (t)+z h (t)]μ h = h(t) [Q (t)+z h (t)]n h N h (t) [Q h (t)+z h (t)] μ (t ). h :h =h Removing the constant tems, (9) can be convete into the following equivalent seve/vm povisioning poblem: min: V c (t)nh(t) [Q h (t)+z h (t)]n hnh(t) (23) s.t.: Nh(t) μ (t )/n h, h H; :h =h Constaints (7), (0).

6 6 The objective function of (23) is equivalent to [ ] N h(t) Vc (t) [Q h (t)+z h (t)]n h an is linea in Nh (t). Fo an efficient solution to (23), we can compute the VM type h as h = agmax [Q h (t)+z h (t)]n h, (24) whee ties ae boken anomly. Thee ae two cases: (i) If Vc (t) [Q (t) +Z h h(t)]n h h=h, the objective function is always non-negative, an Nh (t) s shoul be as small as possible. Hence, only the minimum numbe of seves unning left-ove jobs ae kept on, while the othe seves shoul be tune own in this atacente, i.e., Nh(t) = μ (t )/n h, h H. (25) :h =h (ii) If Vc (t) < [Q (t) +Z h h(t)]n h h=h, all seves in atacente shoul be activate, an except those occupie by left-ove jobs, they shoul povision type h VMs, i.e., Nh(t) = μ (t )/n h, h H,h h, (26) :h =h Nh (t) =N Nh(t). (27) h h Afte Nh (t) s ae ecie, the job scheuling ecisions can be mae base on Eqn. (20)(2)(22). In paticula, in case (i), no new jobs ae scheule onto atacente in t; in case (ii), all newly povisione type-h VMs seve jobs of type h. Dealing with Vaying Job Wokloas. In the stana Lyapunov optimization famewok, minimizing the -slot iftplus-penalty in each time slot can be pove to optimize a time-aveage utility ove the long un, with the citical assumption that all jobs have the equal fixe length, equivalent to one time slot. Decisions mae in one time slot o not influence esouces to be allocate in the subsequent slots [4]. Ou system moel is moe geneal: a type- job scheule in t will occupy a VM fo w time slots, iectly affecting job scheuling an esouce povisioning choices in late times. We novelly make the following esign in ou non-peemptive algoithm, with algoithmic optimality pove in Sec. IV. We goup Γ time slots into a time fame, whee Γ is lage than w max. The above job scheuling an seve/vm povision algoithm vaies slightly epening on which time slot it is unning in (font-en picing an job opping algoithms emain intact): in a time slot t [nγ, (n +)Γ w max ] in the beginning pat of a time fame, the above job scheuling an seve/vm povisioning algoithm emains intact; in a time slot t [(n +)Γ w max +, (n +)Γ ] towas the en of a time fame, the algoithm iffes in that only type- jobs with w (n +)Γ t (i.e., which can be finishe in this time fame), ae consiee in the choice of h : h = agmax :h =h,w (n+)γ t [Q (t)+z (t)], (28) an h is calculate coesponingly by (24). The complete ynamic algoithm, caie out in each time slot by the clou, is summaize in Algoithm. Algoithm Dynamic algoithm in time slot t Input: Q (t), Z (t), μ (t ), c (t), N, n h, V, ɛ, p max,μ,max, G max, Γ ( R, D, h H). Output: p (t), Nh (t), μ (t), G (t) ( R, D, h H ) : fo Each atacente Do 2: Choose the pice by solving the J optimization poblems in (6). 3: if (t mo Γ) [0, Γ w max ] then 4: fo Each VM type h Ho 5: Detemine the type of jobs type-h VMs shoul seve, h, using equation (20). 6: en fo 7: else if (t mo Γ) [Γ w max, Γ ] then 8: fo Each VM type h Ho 9: Detemine the type of jobs type-h VMs shoul seve, h, using equation (28). 0: en fo : en if 2: Detemine type of VMs new configue seves shoul un, h, using equation (24) 3: if [Q h (t)+z h (t)]n h h=h Vc (t) then 4: Keep seves unning leftove jobs on, close all othe seves 5: else 6: Keep seves unning leftove jobs on, configue all othe seves to un type-h VMs, use these type-h VMs to seve type- h h=h jobs. 7: en if 8: Choose the job op numbe accoing to Eqn. (8) 9: Upate the queues Q (t), Z (t) accoing to queue ynamic equations (3) (2). 20: en fo IV. PERFORMANCE ANALYSIS We next analyze the pefomance of Algoithm in tems of queueing elay boun, conitions fo avoiing job opping, an pofit optimality. Detaile poofs can be foun in the elate technical epot []. Theoem. (Queueing Delay Boun) The length of wokloa queue Q is boune by Q,max = Vp,max /w + w a max, an the SLA of jobs can be guaantee by l = Q,max +Z,max ɛ, whee Z,max = V η /w +ɛ is the uppe boun of the length of vitual queue Z. The queue length boun can be pove though inuction. Fo Q, once its queue length excees Vp,max /w,wehave w Q Vp (t) > 0; to minimize (6), ou algoithm takes p (t) >p,max an no new jobs ae amitte in the next time slot. Q will stat to ecease. As the maximum incease in one time slot is w a max, the queue length can not excee Vp,max /w + w a max. Similaly, fo Z, once its queue length exees Vη /w, unseve jobs ae oppe, Z will stat to ecease. As the maximum incease in one time slot is ɛ, the queue length can not excee Vη /w + ɛ. The following theoem states the conitions une which Algoithm guaantees zeo job opping. Theoem 2. (No Job Dop Conitions) If the following two conitions ae satisfie, N n h w ( w )(w max a max + ɛ max ), R, D, R (29)

7 7 V η V cmax w n min +( R w )(w max a max + ɛ max ), R, D. (30) thee is no job opping in any atacente at any time. Hee, c max is the maximum cost fo unning a seve fo one time slot at any atacente. n min is the minimum numbe of VMs a seve in any atacente can host. ɛ max =max{ɛ, R}, a max =max{a max, R}. Conition (29) means that the maximum wokloa euction by scheuling type- jobs once shoul be no smalle than the oveall wokloa accumulate in the coesponing queue since last time when wokloa in the queue was scheule. With conition (29), we can pove that the c sum of queue lengths is uppe-boune by V max + n min ( R w )(wmax a max +ɛ max ), i.e., Q (t)+z (t) V cmax n min + ( R w )(wmax a max + ɛ max ). Une the boun of the sum of queue lengths, conition (30) guaantees Q (t)+z (t) Vη /w. Accoing to job opping ecisions in (8), no job opping woul happen. Hence, to pove the theoem, it is sufficient by showing that the boun of aggegate queue length Q (t) +Z (t) V cmax n min +( R w )(wmax a max + ɛ max ) une conition (29). We pove it by contaiction. Assume Q (t)+z (t) > V cmax n +( min R w )(wmax a max + ɛ max ) une conition (29). As the maximum incease of the sum of queue length in one time slot is w max a max + ɛ max, to achieve a queue length exceeing V cmax n +( min R w )(wmax a max +ɛ max ), R w consecutive time slots ae neee when type- jobs ae not scheule afte the sum just becomes lage than V cmax n. Duing these min R w time slots, as the conition [Q (t) +Z h h(t)]n h > Vcmax > Vc (t), all seves in atacente shoul be tune on in ou algoithm. Hence, othe types of jobs othe than type will be scheule to un among the R w time slots. We can also pove that a type of jobs can not be scheule twice among R w time slots, since if the type of jobs is scheule, its queue length will not be lage than that of type- jobs within the emaining time slots among the R w time slots. This implies that the total numbe of iffeent types of jobs is at least R +, which contaicts the tue total numbe of job types, R. We next pove the pefomance optimality of ou algoithm. Define λ as the vecto of time-aveage wokloas of atacente fo iffeent types of jobs, i.e., λ = lim T T T w t=0 j J (t) Definition (Capacity egion): Une the wokloa aival ate vecto λ, if thee exist peemptive o non-peemptive job scheuling an seve/vm povisioning algoithms that can stabilize all wokloa queues Q (t), R s without job opping o violating the SLA equiements, we say λ is suppotable by atacente. The capacity egion C is the set of all suppotable vectos of wokloa aival ates at atacente. Γ w max λ Definition 2 ((+δ)-optimal Pofit): When the wokloa aival ate vecto at atacente, λ, satisfies (+δ)λ C, the offline optimal time-aveage pofit that is achievable une both peemptive an non-peemptive algoithms without job opping o violating the SLAs is the (+δ)-optimal pofit, enote by P +δ. The following theoem establishes the suppotable wokloa aival ate vecto an the pofit optimality achieve by ou ynamic non-peemptive algoithm compae with the capacity egion an (+δ)-optimal pofit efine above. Note that, - optimal pofit is exactly the offline optimum P fo the pofitmaximization poblem in Eqn. (4) une the SLA constaint with no job ops. Theoem 3. (Pefomance Optimality) When the algoithm an system paametes satisfy conitions (29)(30) an the length of a time fame satisfies Γ > w max, with the assumption that the ynamic powe costs, c (t), D, ae egoic pocesses, thee exists some δ > 0, such that the suppotable wokloa aival ate vecto λ by the nonpeemptive Algoithm satisfies C, the (+δ)γ timeaveage pofit achieve by Algoithm is within a constant Γ gap fom the Γ w -optimum, i.e., max lim π πγ π n=0 (n+)γ t=nγ B V (Γ wmax )(Γ w max ) 2ΓV Γ [(w a max ) 2 +(ɛ ) 2 ] 2V R (Γ wmax )(Γ w max ) 2ΓV wmax Γ N c,max, D E{P (t)} P (+δ)γ Γ w max (3) B N (c,max c,min ) D with B = D R [w a max +2μ,max +ɛ ]μ max, whee the LHS is the time-aveage pofit achieve by Algoithm an the RHS is the (+δ)γ Γ w -optimal pofit minus a constant. max c (,max) an c (,min) ae the maximum an minimum powe consumption costs fo opeating one seve fo one time slot in atacente D. Remak: Ifδ scales own infinitely close to 0, ou algoithm Γ achieves a constant gap fom the Γ w -optimum. Moeove, max if V, Γ an Γ V <, Algoithm has a constant gap fom the -optimum, i.e., the offline optimal pofit achieve by the pofit-maximization poblem in (4). V. PERFORMANCE EVALUATION A. Simulation Setup Geo-istibute Datacentes. We evaluate an IaaS clou opeating thee geo-istibute atacentes locate in thee egions of Noth Ameica: Noth Viginia, Oegon, Nothen Califonia. The efault configuation of the atacentes is as follows. The numbe of seves in each atacente is 000. Thee ae 6 types of vitual machines. Each seve can host

8 8 40 type- VMs, 30 type-2 VMs, 20 type-3 VMs, 5 type- 4 VMs, 0 type-5 VMs o 5 type-6 VMs, which follow the numbes of iffeent types of VMs that a seve on Linoe [6] can host. The powe consumption of each active seve is KW/h an the powe usage effectiveness (PUE) of each atacente is.6. We use eal-wol taces of houly ynamic electicity pices [2] in iffeent egions. Job Types. The clou povies choices among iffeent types of VMs lasting fo iffeent time lengths. The time length, i.e., the numbe of units of wokloa, is chosen among [, 4]. Thee ae 6 4=24types of jobs in total. We emulate uses in thee zones. The utility of uses in zone j when thei type- jobs ae seve by atacente is epesente by a log function U j, = p,max Cj log( + C j ), accoing to the maginal utility iminishing law in economics. C j epesents the iminishing ate of the maginal utility of uses in zone j. The lage C j is, the fewe VMs ae pefee by uses in zone j. Weset[C,C 2,C 3 ]=[2 0 4, 4 0 4, ]. The maximum acceptable pice fo a type- job in atacente is set equal to the maximum powe cost fo completing the job. Fo compaison puposes, we implement two othe stategies: () Static picing with the same job scheuling an seve povisioning stategies as in Algoithm, compaison against which will show the avantage of ynamic picing ove static picing (such as the picing stategy in Amazon EC2 s on-eman instance maket). (2) A heuistic picing an job scheuling algoithm, which opeates as follows in each time slot: (a) Picing. In each atacente, the wokloa of each type of jobs is still maintaine in a wokloa queue Q. When the oveall amount of wokloas in Q is smalle than a theshol S, the pice chage to a type- job is set to the smallest use s willingness-to-pay that will not make newly accepte wokloas excee the queue theshol S at this time slot; when the oveall amount of wokloas is equal to S, the maximum pice p,max is set an no new jobs ae accepte. (b) Job an Seve Scheuling. The heuistic calculates the aveage pice fo one unit of wokloa, chage to jobs in each wokloa queue, an multiplies the aveage pice fo queue Q by the numbe of type-h VMs that one seve can host in each atacente, to obtain the pofit fo configuing one seve to un type-h VMs in each atacente. Each atacente configues seves to un the type of VMs that achieves the lagest pofit an scheules the coesponing type of jobs. The cost fo unning one seve in the cuent time slot in each atacente is also calculate. If the lagest pofit fo unning one seve is lage than the cost in one atacente, the coesponing type of jobs ae scheule to seves in the atacente; othewise, jobs ae not scheule. The heuistic picing an job scheuling is an algoithm without optimization fo pofit. B. Pofit an Cost We un ou ynamic algoithm fo T = 240 time slots with paametes V =5 0 5, ɛ =50 w, η = 000 p,max, an Γ = 00w max. Fig. pesents the pofit, evenue, powe cost TABLE II PROFIT UNDER DIFFERENT STATIC PRICES Potion Pofit an penalty ue to job ops in the clou in each time slot. The nomalize value is calculate by iviing the oiginal value in each time slot by the maximum evenue within this peio. We obseve that a stable pofit is achieve by ou ynamic algoithm. It can be seen that no penalty is incue, i.e., no job op occus, which veifies ou analysis on no job op pesente in Sec. IV. Fig. 2 shows the pofits achieve by the thee algoithms espectively. The heuistic picing an scheuling algoithm sets S to be equal to the maximum numbe of type-h VMs that atacente can povie, ivie by the numbe of job types equiing type- VMs. The static picing fixes the pices fo each type of jobs in each atacente above the lowe boun of the powe cost fo completing such a job. Table II gives the pofit achieve by the static picing algoithm, by setting the static pice to be iffeent popotions of the maximum powe cost. Fom the table we see that when the static pice is 0. of the maximum powe cost, the pofit is lage than in othe cases. Hence, we use 0. of the maximum powe cost as the static pice, in the compaisons with othe algoithms in Fig. 2. The nomalize pofit is calculate by iviing the pofit in each time slot by the maximum pofit in one time slot within this peio among the thee algoithms. We can obseve that ou ynamic picing algoithm outpefoms the othe two algoithms, an achieves stable pofit ove time. Fig.. Nomalize value Nomalize pofit Revenue Pofit Powe cost Penalty Time slot Revenue, powe cost, pofit an penalty in each time slot Dynamic picing Heuistic algoithm Static picing Time slot Fig. 2. Compaison of pofits among iffeent algoithms. C. Impact of V an Γ Fig. 3 an 4 futhe illustate how the time-aveage pofit achieve by ou algoithm vaies with iffeent choices of V

9 9 an Γ, espectively. The time-aveage pofit is nomalize by being ivie by the time-aveage pofit une paametes V =5 0 5, Γ = 00w max. Fig. 3 shows that as V inceases, the time-aveage pofit inceases, veifying the ole of V given in Theoem 3. Γ is the numbe of time slots in a time fame. Fig. 4 suggests that, when Γ is lage than 0w max, its value has no substantial impact on pofits, evealing the fact that ou two-time-scale ynamic algoithm is not sensitive to the exact length of time fames. As V inceases to infinity an Γ is lage enough, the time-aveage pofit is abitaily close to a constant gap fom the offline optimum. Time aveage pofits Γ=0 w max Γ=50 w max Γ=00 w max V Value x 0 5 Fig. 3. Time-aveage pofits une iffeent values of V. Time aveage pofits V= 0 5 V=3 0 5 V= Γ Value Fig. 4. Time-aveage pofits une iffeent values of Γ. VI. RELATED WORK A numbe of stuies apply auctions to pice computing esouces in a clou system [3] [4] [5]. Wang et al. [3] moel VM picing as a multi-unit combinatoial auction, which is execute oun by oun without consieing that uses may occupy a VM fo moe than one ecision inteval. Wang et al. [4] moel a ynamic auction whee bies may equest to occupy a VM fo moe than one ecision inteval, such that the auction in one oun is coelate with that in anothe oun. Zhang et al. [5] povie a tuthful online auction famewok to pocess uses instantaneous an heteogeneous bis fo esouces. They both assume that the capacity of the clou is fixe, without aessing seve povisioning in the system. Anothe goup of wok stuies clou esouce scheuling une given picing stategies [6] [7]. Wang et al. [6] stuy how a clou shoul allocate its esouces between the on-eman maket an the auction maket. Zhang et al. [7] popose a ynamic scheuling an consoliation mechanism that allocates VM esouces to each spot maket to maximize the clou povie s total evenue. Diffeently, ou wok jointly moels ynamical esouce picing an scheuling. Most wok that apply the Lyapunov optimization famewok fo wokloa scheuling in clou systems implicitly assume wokloa that woul only occupy the souces within the uation of one ecision inteval [5] [8]. We ae awae of only one stuy by Magului et al. [9] that investigates the scheuling of vaiable-length jobs in clou systems, using Lyapunov optimization. Thei scheuling aims to stabilize queues in the system, while we taget close-to-offline-optimal pefomance in pofit maximization. VII. CONCLUSION This pape poposes an online algoithm fo joint VM picing, job scheuling an seve povisioning in a clou consisting of geo-istibute atacentes. The algoithm takes into consieation the case that the execution time of each job may be longe than the inteval of online ecisions. The lowe boun of the time-aveage pofit achieve by the algoithm is poven to appoach the offline optimum minus a constant, which iminishes when appopiate paametes ae chosen. We also analyze the conitions fo the clou not to op jobs ue to violating the elay constaints. Empiical stuies une ealistic settings valiate ou theoetical esults. REFERENCES [] Winows Azue, [2] AMAZON EC2, [3] GCE, [4] M. J. Neely, Stochastic Netwok Optimization with Application to Communication an Queueing Systems. Mogan & Claypool, 200. [5] Y. Yao, L. B. Huang, A. Shama, L. Golubchik, an M. Neely, Data Centes Powe Reuction: A Two Time Scale Appoach fo Delay Toleant Wokloas, in Poc. of INFOCOM, Mach 202. [6] LINODE, [7] M. R. Hines, U. Deshpane, an K. Gopalan, Post-copy live migation of vitual machines, ACM SIGOPS Opeating Systems Review, vol. 43, no. 3, p. 4, [Online]. Available: [8] A. Geenbeg, J. Hamilton, D. A. Maltz, an P. Patel, The Cost of a Clou: Reseach Poblems in Data Cente Netwoks, in ACM SIGCOMM C. C. R., vol. 39, no., Januay 2009, pp [9] M. Neely, Oppotunistic Scheuling with Wost Case Delay Guaantees in Single an Multi-Hop Netwoks, in Poc. of INFOCOM, Ma [0] L. Geogiais, M. J. Neely, an L. Tassiulas, Resouce Allocation an Coss-Laye Contol in Wieless Netwoks, 2006, vol., no.. [] J. Zhao, H. Li, C. Wu, Z. Li, Z. Zhang, an F. Lau, Dynamic Picing an Pofit Maximization fo Clous with Geo-istibute Datacentes, Tech. Rep., [2] Feeal Enegy Regulatoy Commission, [3] Q. Wang, K. Ren, an X. Meng, When Clou Meets ebay: Towas Effective Picing fo Clou Computing, in IEEE INFOCOM, 202. [4] W. Wang, B. Liang, an B. Li, Revenue Maximization with Dynamic Auctions in IaaS Clou Makets, in Poc. of IWQoS, 203. [5] H. Zhang, B. Li, H. B. Jiang, F. M. Liu, A. V. Vasilakos, an J. C. Liu, A Famewok fo Tuthful Online Auctions in Clou Computing with Heteogeneous Use Demans, in IEEE INFOCOM, Ap [6] W. Wang, B. Li, an B. Liang, Towas Optimal Capacity Segmentation with Hybi Clou Picing, in Poc. of ICDCS, 202. [7] Q. Zhang, E. Gses, R. Boutaba, an J. Xiao, Dynamic esouce allocation fo spot makets in clous, in Poc. of Hot-ICE, 20. [8] K. Le, J. Zhang, J. Meng, R. Bianchini, Y. Jaluia, an T. Nguyen, Reucing Electicity Cost Though Vitual Machine Placement in High Pefomance Computing Clous, in Supecomputing, Nov. 20. [9] S. T. Magului, R. Sikant, an L. Ying, Stochastic Moels of Loa Balancing an Scheuling in Clou Computing Clustes, in Poc. of INFOCOM, Mach 202.