This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and

Transcription

1 Ths artcle appeared n a ournal publshed by Elsever. The attached copy s furnshed to the author for nternal non-commercal research educaton use, ncludng for nstructon at the authors nsttuton sharng wth colleagues. Other uses, ncludng reproducton dstrbuton, or sellng or lcensng copes, or postng to personal, nsttutonal or thrd party webstes are prohbted. In most cases authors are permtted to post ther verson of the artcle (e.g. n Word or Tex form) to ther personal webste or nsttutonal repostory. Authors requrng further nformaton regardng Elsever s archvng manuscrpt polces are encouraged to vst:

2 Theoretcal Computer Scence 496 (203) 3 24 Contents lsts avalable at ScVerse ScenceDrect Theoretcal Computer Scence ournal homepage: Economc models for cloud servce markets: Prcng Capacty plannng Ranan Pal, Pan Hu Unversty of Southern Calforna, USA Deutsch Telekom Laboratores, Germany artcle Keywords: Cloud markets Competton Nash equlbrum Capacty Sngle-ter Mult-ter nfo abstract Cloud computng s a paradgm that has the potental to transform revolutonalze the next generaton IT ndustry by makng software avalable to end-users as a servce. A cloud, also commonly known as a cloud network, typcally comprses of hardware (network of servers) a collecton of softwares that s made avalable to end-users n a payas-you-go manner. Multple publc cloud provders (e.g., Amazon) co-exstng n a cloud computng market provde smlar servces (software as a servce) to ts clents, both n terms of the nature of an applcaton, as well as n qualty of servce (QoS) provson. The decson of whether a cloud hosts (or fnds t proftable to host) a servce n the long-term would depend ontly on the prce t sets, the QoS guarantees t provdes to ts customers, the satsfacton of the advertsed guarantees. In the frst part of the paper, we devse analyze three nter-organzatonal economc models relevant to cloud networks. We formulate our problems as non cooperatve prce QoS games between multple cloud provders exstng n a cloud market. We prove that a unque pure strategy Nash equlbrum (NE) exsts n two of the three models. Our analyss paves the path for each cloud provder to know what prces QoS level to set for end-users of a gven servce type, such that the provder could exst n the cloud market. A cloud provder servces end-user requests on behalf of cloud customers, due to the uncertanty n user dems over tme, tend to over-provson resources lke CPU, power, memory, storage, etc., n order to satsfy QoS guarantees. As a result of over-provsonng over long tmescales, server utlzaton s very low the cloud provders have to bear unnecessarly wasteful costs. In ths regard, the prce QoS levels set by the CPs drve the end-user dem, whch plays a maor role n CPs estmatng the mnmal capacty to meet ther advertsed guarantees. By the term capacty, we mply the ablty of a cloud to process user requests,.e., number of user requests processed per unt of tme, whch n turn determne the amount of resources to be provsoned to acheve a requred capacty. In the second part of ths paper, we address the capacty plannng/optmal resource provsonng problem n sngle-tered mult-tered cloud networks usng a technoeconomc approach. We develop, analyze, compare models that cloud provders can adopt to provson resources n a manner such that there s mnmum amount of resources wasted, at the same tme the user servce-level/qos guarantees are satsfed. Publshed by Elsever B.V.. Introducton Cloud computng s a type of Internet-based computng, where shared resources, hardware, software, nformaton are provded to end-users n an on dem fashon. It s a paradgm that has the potental to transform revolutonalze Correspondng author at: Unversty of Southern Calforna, USA. E-mal addresses: rpal@usc.edu (R. Pal), pan.hu@telekom.de (P. Hu) /$ see front matter. Publshed by Elsever B.V. do:0.06/.tcs

3 4 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) 3 24 the IT ndustry by makng software avalable to end-users as a servce []. A publc cloud typcally comprses of hardware (network of servers) a collecton of softwares that s made avalable to the general publc n a pay-as-you-go manner. Typcal examples of companes provdng publc clouds nclude Amazon, Google, Mcrosoft, E-Bay, commercal banks. Publc cloud provders usually provde Software as a Servce (SaaS), Platform as a Servce (PaaS), Infrastructure as a Servce (IaaS). The advantage of makng software avalable as a servce s three-fold [], () the servce provders beneft from smplfed software nstallaton, mantenance, centralzed versonng, (2) end-users can access the software n an anytme anywhere manner, can store data safely n the cloud nfrastructure, do not have to thnk about provsonng any hardware resource due to the lluson of nfnte computng resources avalable on dem, (3) end-users can pay for usng computng resources on a short-term bass (e.g., by the hour or by the day) can release the resources on task completon. Smlar beneft types are also obtaned by makng both, platform as well as nfrastructure avalable as servce. Cloud economcs wll play a vtal role n shapng the cloud computng ndustry of the future. In a recent Mcrosoft whte paper ttled Economcs of the Cloud, t has been stated that the computng ndustry s movng towards the cloud drven by three mportant economes of scale: () large data centers can deploy computatonal resources at sgnfcantly lower costs than smaller ones, (2) dem poolng mproves utlzaton of resources, (3) mult-tenancy lowers applcaton mantenance labor costs for large publc clouds. The cloud also provdes an opportunty to IT professonals to focus more on technologcal nnovaton rather than thnkng of the budget of keepng the lghts on. The economcs of the cloud can be thought of havng two dmensons: () ntra-organzaton economcs (2) nter-organzaton economcs. Intraorganzaton economcs deals wth the economcs of nternal factors of an organzaton lke labor, power, hardware, securty, etc., whereas nter-organzaton economcs refers to the economcs of market competton factors between organzatons. Examples of some popular factors are prce, QoS, reputaton, customer servce. In ths paper, we focus on nterorganzatonal economc ssues. Multple publc cloud provders (e.g., Amazon, Google, Mcrosoft, etc.,) co-exstng n a cloud computng market provde smlar servces (software as a servce, e.g., Google Docs Mcrosoft Offce Lve) to ts clents, both n terms of the nature of an applcaton, as well as n qualty of servce (QoS) provson. The decson of whether a cloud hosts (or fnds t proftable to host) a servce n the long-term would (amongst other factors) depend ontly on the prce t sets, the QoS guarantees t provdes to ts customers, the satsfacton of the advertsed guarantees. Settng hgh prces mght result n a drop n dem for a partcular servce, whereas settng low prces mght attract customers at the expense of lowerng cloud provder profts. Smlarly, advertsng satsfyng hgh QoS levels would favor a cloud provder (CP) n attractng more customers. The prce QoS levels set by the CPs thus drve the end-user dem, whch, apart from determnng the market power of a CP also plays a maor role n CPs estmatng the mnmal resource capacty to meet ther advertsed guarantees. By the term capacty, we mply the ablty of a cloud to process user requests,.e., number of user requests processed per unt of tme. The estmaton problem s an mportant challenge n cloud computng wth respect to resource provsonng because a successful estmaton would prevent CPs to provson for the peak, thereby reducng resource wastage. The competton n prces QoS amongst the cloud provders entals the formaton of non-cooperatve games amongst compettve CPs. Thus, we have a dstrbuted system of CPs (players n the game), where each CP wants to maxmze ts own profts would tend towards playng a Nash equlbrum 2 (NE) strategy (.e., each CP would want to set the NE prces QoS levels), whereby the whole system of CPs would have no ncentve to devate from the Nash equlbrum pont,.e., the vector of NE strateges of each CP. However, for each CP to play a NE strategy, the latter should mathematcally exst. In the frst part of the paper, we address the mportant problem of Nash Equlbrum characterzaton of dfferent types of prce QoS games relevant to cloud networks, ts propertes, practcal mplementablty (convergence ssues), the senstvty analyss of NE prce/qos varatons by any CP on the prce QoS levels of other CPs. Our problem s mportant from a resource provsonng perspectve as mentoned n the prevous paragraph, apart from t havng obvous strategc mportance on CPs n terms of sustenance n the cloud market. In the second part of our paper we develop analyze models that wll be useful to cloud provders to provson resources n a manner such that there s mnmum amount of resources wasted, at the same tme the user servce-level/qos guarantees are satsfed... Related work In regard to market competton drven network prcng, there exsts research work n the doman of multple ISP nteracton tered Internet servces [2,3], as well as n the area of resource allocaton Internet congeston management [4 6]. However, the market competton n our work relates to optmal capacty plannng resource provsonng n clouds. There s the semnal work by Songhurst Kelly [7] on prcng schemes based on QoS requrements of users. Ther work address mult-servce scenaros derve prcng schemes for each servce based on the QoS A cloud provder generally gets requests from a cloud customer, whch n turn accepts requests from Internet end-users. Thus, typcally, the clents/customers of a cloud provder are the cloud customers. However, for modelng purposes, end-users could also be treated as customers. (See Secton 2.) 2 A group of players s n Nash equlbrum f each one s makng the best decson (strategy) that he or she can, takng nto account the decsons of the others.

4 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) requrements for each, n turn bwdth reservatons. Ths work resembles ours to some extent n the sense that the prce QoS determned can determne optmal bwdth provsons. However, t does not account for market competton between multple provders only focus on a sngle servce provder provdng multple servces,.e., the paper addresses an ntra-organzaton economcs problem. However, n ths paper, we assume sngle-servce scenaros by multple servce provders. In a recent work [8], the authors propose a queueng drven game-theoretc model for prce QoS competton amongst multple servce provders. The work analyzes a duopolstc market between two servce provders, where provders frst fx ther QoS guarantees then compete for prces. Our work extends the latter cted work n the followng aspects: () we generalze our model to ncorporate n servce provders, (2) we address two addtonal game models whch are of practcal mportance,.e., prce QoS smultaneous competton prces fxed frst, followed by QoS guarantees competton, (3) we provde an effcent technque to compute multple equlbra n games, (4) our models explctly characterze percentle performance of parameters, whch s specfc to cloud networks provsonng resources on a percentle bass. We also want to emphasze the fact that research on prce/qos competton amongst organzatons s not new n the economcs doman. However, n ths paper we model networkng elements n prce/qos games va a queueng theoretc approach analyze certan prce/qos games that are manly characterstc of Internet servce markets. Recent research efforts on cloud resource provsonng have devsed statc dynamc provsonng schemes. Statc provsonng [9,20] s usually conducted offlne occurs on monthly or seasonal tmescales, 3 whereas dynamc provsonng [2,22] dynamcally adusts to workload fluctuatons over tme. In both the statc the dynamc case, vrtual machne (VM) szng [] s dentfed as the most mportant step, where VM szng refers to the estmaton of the amount of resources to be allocated to a VM or ontly to many VMs [23]. However, none of the above cted works have accounted for external factors such as cloud provder prce competton, n determnng the optmal capacty of a cloud provder for a gven tme-slot. Market competton between cloud provders s a vtal factor n capacty plannng because cloud provders set prces to prmarly to make profts the prces they set nfluence dems from end-users, user dems drve the provsonng of optmal capactes. Other factors lke schedulng polces (e.g., FCFS, Processor Sharng, etc.) employed by cloud provders, as well as the number of ters a web applcaton needs for servce, also contrbute to optmal capacty provsonng. Recent works on cloud network provsonng have accounted for parameters lke schedulng mult-ter servces [24], but do not provde any analytcal results on the mpact of these parameters on optmally provsoned capacty, nor do they evaluate the optmal provsoned capacty. In contrast wth exstng approaches, we take a techno-economc approach to evaluatng the optmal provsoned capacty provde theoretcal nsghts for our problem. Our optmal provsoned capacty s metrczed by the number of user requests processed per unt of tme. However, ths noton of capacty can be mapped to physcal resource capacty metrcs lke bwdth, CPU, etc. Our proposed models am to focus on how certan techncal economc parameters nfluence optmal provsoned capacty of a cloud provder, as well as other competng cloud provders, whch s mportant when t comes to network desgn..2. Contrbutons statement Our proposed theory analyzes a few basc nter-organzatonal economc models through whch cloud servces could be prced under market competton. The evoluton of commercal publc cloud servce markets s stll n ts ncepton. However, wth the ganng popularty of cloud servces, we expect a bg surge n publc cloud servces competton n the years to come. The models proposed n ths paper take a substantal step n hghlghtng relevant models to the cloud networkng communty for them adopt so as to approprately prce current future cloud servces. In practce, scenaros of prce /or QoS competton between organzatons exst n the moble network servces ISP markets. For example, AT&T Verzon are competng on servce,.e., Verzon promses to provde better coverage to moble users than AT&T, thereby ncreasng ts propensty to attract more customers. Smlarly, prce competton between ISPs always exsted for provdng broadb servces at a certan gven bwdth guarantee. Regardng our work, we also want to emphasze () we do not make any clams about our models beng the only way to model nter-organzatonal cloud economcs 4 (2) there s a dependency between ntra-organzatonal nter-organzatonal economc factors, whch we do not account n ths paper due to modelng smplcty. However, through our work, we defntely provde readers wth a concrete modelng ntuton to go about addressng problems n cloud economcs. To the best of our knowledge, we are the frst to provde an analytcal model on nter-organzatonal cloud economcs. Our Contrbutons We make the followng contrbutons n ths paper.. We formulate a separable end-user dem functon for each cloud provder w.r.t. to prce QoS levels set by them derve ther ndvdual utlty functons (proft functon). We then defne the varous prce QoS games that we analyze n the paper. (See Secton 2.) 2. We develop a model where the QoS guarantees provded by publc CPs to end-users for a partcular applcaton type are pre-specfed fxed, the cloud provders compete for prces. We formulate a non-cooperatve prce game amongst 3 Several cloud management softwares lke VMWare Capacty Planner, CapactyIQ, IBM WebSphere CloudBurst adopt ths functonalty. 4 We only model prce QoS as parameters. One could choose other parameters (n addton to prce QoS, whch are essental parameters) a dfferent analyss mechansm than ours to arrve at a dfferent model.

5 6 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) 3 24 the players (.e., the cloud provders) prove that there exsts a unque Nash equlbrum of the game, that the NE could be practcally computed (.e., t converges). (See Secton 3.) 3. We develop a non-cooperatve game-theoretc model where publc cloud provders ontly compete for the prce QoS levels related to a partcular applcaton type. We show the exstence convergence of Nash equlbra. (See Secton 4.) As a specal case of ths model, we also analyze the case where prces charged to Internet end-users are pre-specfed fxed, the cloud provders compete for QoS guarantees only. The models mentoned n contrbutons 3 4 drve optmal capacty plannng resource provsonng n clouds, apart from maxmzng CP profts. (See Secton 4.) 4. We conduct a senstvty analyss on varous parameters of our proposed models, study the effect of changes n the parameters on the equlbrum prce QoS levels of the CPs exstng n a cloud market. Through a senstvty analyss, we nfer the effect of prce QoS changes of cloud provders on ther respectve profts, as well as the profts of competng CPs. (See Sectons 3 4.) 5 5. We develop an optmzaton framework for sngle-tered mult-tered cloud networks to compute the optmal provsoned capacty once the equlbrum prce QoS levels for each CP have been determned. (See Secton 5.) 2. Problem setup We consder a market of n competng cloud provders, where each provder servces applcaton types to end-users at a gven QoS guarantee. We assume that end-users are customers of cloud provders n an ndrect manner,.e., Internet end-users use onlne softwares developed by companes (cloud customers), that depend on cloud provders to servce ther customer requests. Each CP s n competton wth others n the market for servces provded on the same type of applcaton w.r.t functonalty QoS guarantees. For example, Mcrosoft Google mght both serve a word processng applcaton to end-users by provdng smlar QoS guarantees. Here, the word processng applcaton represents a partcular type. For a gven applcaton type, we assume that each end user sgns a contract wth a partcular CP for a gven tme perod, 6 wthn that perod t does not swtch to any other CP for gettng servce on the same applcaton type. Regardng contracts between a CP ts end-users, we assume that a cloud customer forwards servce requests to a cloud provder on behalf of end-users, who sgn up wth a cloud customer (CC) for servce. The CP charges ts cloud customer, who n turn charges ts end-users. We approxmate ths two-step chargng scheme by modelng a vrtual one-step scheme, where a CP charges end-users drectly. 7 In a gven tme perod, each CP postons tself n the market by selectng a prce p a QoS level s related to a gven applcaton type. Throughout the paper, we assume that the CPs compete on a sngle gven type. 8 We defne s as the dfference between a benchmark response tme upper bound, rt, the actual response tme rt,.e., s = rt rt. For example, f for a partcular applcaton type, every CP would respond to an end-user request wthn 0 s, rt = 0. The response tme rt may be defned, ether n terms of the expected steady state response tme,.e., rt = E(RT ), or n terms of -percentle performance, rt ( ), where 0 < <. Thus, n terms of -percentle performance, 9 P(RT < rt ( )) =. We model each CP as an M/M/ queueng system, where end-user requests arrve as a Posson process wth mean rate, gets servced at a rate µ. We adopt an M/M/ queueng system because of three reasons: () queueng theory has been tradtonally used n request arrval servce problems, (2) for our problem, assumng an M/M/ queueng system ensures tractable analyses procedures that entals dervng nce closed form expressons helps underst system nsghts n a non-complex manner, wthout sacrfcng a great deal n capturng the real dynamcs of the actual arrval departure process, (3) the Markovan nature of the servce process helps us generalze expected steady state analyss percentle analyss together. Accordng to the theory of M/M/ queues, we have the followng stard results [7]. rt = µ, rt ( ) = ln( ), µ ( ) µ = +, rt () (2) (3) 5 We study Nash equlbrum convergence as t proves the achevablty of an equlbrum pont n the market. We emphasze here that the exstence of Nash equlbrum does not mply achevablty as t may take the cloud market an eternty to reach equlbrum, even though there may exst one theoretcally. 6 In ths paper, the term tme-perod refers to the tme duraton of a contract between the CP end-users. 7 We assume here that prces are negotated between the CP, CC, end-users there s a vrtual drect prce chargng connecton between the CP ts end-users. We make ths approxmaton for modelng smplcty. 8 In realty, each CP may n general servce several applcaton types concurrently. We do not model ths case n our paper leave t for future work. The case for sngle applcaton types gves nterestng results, whch would prove to be useful n analyzng the multple concurrent applcaton type scenaro. 9 As an example, n cloud networks we often assocate provsonng power accordng to the 95th percentle use. Lkewse, we could also provson servce capacty by accountng for percentle response tme guarantees.

6 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) µ ( ) = + ln( ) rt ( ). Eqs. (2) (4) follow from the fact that for M/M/ queues, P(RT < rt ( )) = = e (µ rt ( )). Wthout loss of generalty, n subsequent sectons of ths paper, we conduct our analyss on expected steady state parameters. As mentoned prevously, due to the Markovan nature of the servce process, the case for percentles s exactly smlar to the case for expected steady state analyss, the only dfference n analyss beng due to the constant, ln( ). Thus, all our proposed equlbrum related results hold true for percentle analyss as well. Each cloud provder ncurs a fxed cost c per user request served a fxed cost per unt of servce capacty provsoned. c arses due to the factor n Eq. (3) arses due to the factor n the same equaton. In ths sense, our rt QoS-dependent prcng models are queueng-drven. A cloud provder charges pr to servce each end-user request, where pr [pr mn, pr max ]. It s evdent that each CP selects a prce that results n t accrung a non-negatve gross proft margn. The gross proft margn for CP s gven as pr c, where c + s the margnal cost per unt of end-user dem. Thus, the prce lower bound, pr mn, for each CP s determned by the followng equaton. pr mn = c +, 8 =,...,n. We defne the dem of any CP,, as a functon of the vectors pr = (pr,...,pr n ) s = (s,...,s n ). Mathematcally, we express the dem functon as = (pr, s) = x (s ) y pr (s ) + pr, (6) 6= 6= where x (s ) s an ncreasng, concave, thrce dfferentable functon n s satsfyng the property of non-ncreasng margnal returns to scale,.e., equal-szed reductons n response tme results n progressvely smaller ncreases n end-user dem. The functons are assumed to be non-decreasng dfferentable. A typcal example of a functon fttng x (s ) (s ) s a logarthmc functon. We model Eq. (6) as a separable functon of prce QoS vectors, for ensurng tractable analyses as well as for extractng ndependent effects of prce QoS changes on the overall end-user dem. Intutvely, Eq. (6) states that QoS mprovements by a CP result n an ncrease n ts end-user dem, whereas QoS mprovements by other compettor CPs result n a decrease n ts dem. Smlarly, a prce ncrease by a CP results n a decrease n ts enduser dem, whereas prce ncreases by other competng CPs result n an ncrease n ts dem. Wthout loss of practcal generalty, we also assume () a unform ncrease n prces by all n CPs cannot result n an ncrease n any CP s dem volume, (2) a prce ncrease by a gven CP cannot result n an ncrease n the market s aggregate end-user dem. Mathematcally, we represent these two facts by the followng two relatonshps. y > 6=, =,...,n (4) (5) (7) y > 6=, =,...,n. (8) The long run average proft for CP n a gven tme perod, assumng that response tmes are expressed n terms of expected values, s a functon of the prce QoS levels of CPs, s gven as P (pr, s) = (pr c ) rt s, 8. (9) The proft functon for each CP acts as ts utlty/payoff functon when t s nvolved n prce QoS games wth other competng CPs. We assume n ths paper that the proft functon for each CP s known to other CPs, but none of the CPs know the values of the parameters that other competng CPs adopt as ther strategy. Problem Statement: Gven the proft functon for each CP (publc nformaton), how would each advertse ts prce QoS values (wthout negotatng wth other CPs) to end-users so as to maxmze ts own proft. In other words, n a compettve game of profts played by CPs, s there a stuaton where each CP s happy wth ts (prce, QoS) advertsed par does not beneft by a postve or negatve devaton n the values of the advertsed par. In ths paper, we study games nvolvng prce QoS as the prmary parameters,.e., we characterze analyze the exstence, unqueness, convergence of Nash equlbra. Our prmary goal s to compute the optmal prce QoS levels offered by CPs to ts end-users under market competton. Our analyss paves the path for each cloud provder to () know what prce QoS levels to set for ts clents (end-users) for a gven applcaton type, such that t could exst n the cloud market, (2) practcally dynamcally provson approprate capacty for satsfyng advertsed QoS guarantees, by takng advantage of the property of vrtualzaton n cloud networks. The property of vrtualzaton entals each CP to allocate optmal resources dynamcally n a fast manner to servce end-user requests. Usng our prcng framework, n each tme perod, cloud provders set the approprate prce QoS levels after competng n a game; the resultng prces drve enduser dem; the CPs then allocate optmal resources to servce dem.

7 8 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) 3 24 Table Lst of symbols ther meanng. Symbol U = P pr pr pr c rt rt C s s s x () () Meanng Utlty functon of CP Prce charged by CP per end-user Prce vector of CPs Nash equlbrum prce vector Cost ncurred by CP to servce each user Arrval rate of end-users to CP Cost/unt of capacty provsonng by CP Response tme upper bound guarantee Response tme guarantee by CP Capacty cost of CP for provsonng ts user dems Percentle parameter QoS level guarantee provded by CP to ts users QoS vector of CPs Nash equlbrum QoS vector Increasng, concave, a thrce dfferentable functon Non-decreasng dfferentable functon Remark. We decded to not analyze a compettve market,.e., where CPs are prce/qos takng a Walrasan equlbrum results when dem equals supply, because a compettve market analyss s manly applcable when the resources traded by an organzaton are neglgble wth respect to the total resource n the system [9,0]. In a cloud market ths s defntely not the case as there are a few cloud provders so the resource traded by one s not neglgble wth respect to the total resources traded n the system. Therefore we analyze olgopolstc markets where CPs are prce/qos antcpatng. We consder the followng types of prce QoS game models n our work.. CP QoS guarantees are pre-specfed; CPs compete wth each other for prces, gven QoS guarantees. (Game ) 2. CPs compete for prce QoS smultaneously. (Game 2) 3. CP prce levels are pre-specfed; CPs compete for QoS levels. (Game 3). Game 3 s a specal case of Game 2 n Secton 4, we wll show that t s a Game 2 dervatve. Lst of Notatons: For reader smplcty, we provde a table of most used notatons related to the analyss of games n ths paper (see Table ). 3. Game prce game In ths secton, we analyze the game n whch the QoS guarantees of CPs are exogenously specfed the CPs compete for prces. Game descrpton Players: Indvdual cloud provders; Game type: non-cooperatve,.e., no nteracton between CPs; Strategy space: choosng a prce n range [pr mn, pr max ]; Player goal: to maxmze ts ndvdual utlty U = P. Our frst goal s to show that ths game has a unque prce Nash equlbrum, pr (an nstance of vector pr), whch satsfes the followng frst = y (pr c ) +, 8, (0) whch n matrx notaton can be represented as M pr = x(s) + z, () where M s an n n matrx wth M = 2y, M =, 6=, where z = y (c + ). We have the followng theorem corollary regardng equlbrum results for our game. The readers are referred to the Appendx for the proofs. Theorem. Gven that the QoS guarantees of CPs are exogenously specfed, the prce competton game has a unque Nash equlbrum, pr, whch satsfes Eq. (). The Nash equlbrum user dem,, for each CP evaluates to y (pr c ), the Nash equlbrum profts, P, for each CP s gven by y (pr c ) 2. rt s Corollary. (a) pr are ncreasng decreasng respectvely n each of the parameters {c,, =, = (M ) x 0 P (s ) l6= (M ) l x 0 l (s ).

8 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) Corollary mples that () under a larger value for CP s degree of postve externalty, t s wllng to make a bolder prce adustment to an ncrease n any of ts cost parameters, thereby mantanng a larger porton of ts orgnal proft margn. The reason s that competng CPs respond wth larger prce themselves, (2) there exsts a crtcal value 0 apple s 0 apple rt such that as CP ncreases ts QoS level, pr are ncreasng on the nterval [0, s 0 ), decreasng n the nterval [s0, rt). Senstvty analyss: We know the followng = 2y (pr @s From t we can nfer that CP s proft ncreases as a result of QoS level mprovement by a competng CP f only f the QoS level mprovement results n an ncrease n CP s prce. Ths happens when P ncreases on the nterval [0, s 0 ] decreases on the remanng nterval (s 0, rt]. In regard to proft varaton trends, on ts own QoS level mprovement, a domnant trend for a CP s not observed. However, we make two observatons based on the holdng of the followng = 2y (pr @s (2) (rt s ) 2. (3) If a CP ncreases ts QoS level from 0 to a postve value ths results n ts prce decrease, s equlbrum profts become a decreasng functon of ts QoS level at all tmes. Thus, n such a case s better off provdng mnmal QoS level to ts customers. However, when CP s QoS level ncreases from 0 to a postve value resultng n an ncrease n ts prce charged to customers, there exsts a QoS level s b such that the equlbrum proft alternates arbtrarly between ncreasng decreasng n the nterval [0, s b ), decreases when s s b. Convergence to Nash equlbra: Snce the prce game n queston has a unque optmal Nash equlbra, t can be found by solvng the system of frst order = 0 for all. Remark. It s true that the exstence of NE n convex games s not surprsng n vew of the general theory, but what s more mportant s whether a realstc modelng of our problem at h results n a convex game. Once we can establsh that our model results n a convex game, we have a straghtforward result of the exstence of NE from the game theory lterature. Ths s exactly what we do n the paper,.e., to show that our model s realstc ndeed leads to a convex game thus leadng further to the exstence of NE. 4. Game 2 prce QoS game In ths secton, we analyze the game n whch the CPs compete for both, prce as well as QoS levels. In the process of analyzng Game 2, we also derve Game 3, as a specal case of Game 2, state results pertanng to Game 3. Game descrpton Players: ndvdual cloud provders; Game type: non-cooperatve,.e., no nteracton between CPs; Strategy space: prce n range [pr mn, pr max ] QoS level s ; Player goal: to maxmze ts ndvdual utlty U = P. We have the followng theorem regardng equlbrum results. r Theorem 2. Let rt apple 3 4y (x 0 ), where y = mn 2 y, = mn, x 0 = max x 0 (0). There exsts a Nash equlbrum (pr, s ), whch satsfes the followng system = y (pr c ) + = 0, 8, (4) satsfes the condton that ether s (pr ) s the unque root of x 0 (s )(pr c ) = (rt s ) f pr 2 c + ( + rt 2 x 0 or (0)) s (pr ) = 0 otherwse. Conversely, any soluton of these two equatons s a Nash equlbrum. Senstvty analyss: We know that s (pr ) depends on x 0 (s ) pr. Thus, from the mplct functon theorem [] we nfer that the QoS level of CP ncreases wth the ncrease n ts Nash equlbrum prce. We have the followng relatonshp for pr > c + ( + rt 2 x 0 ), (0) s 0 (pr x 0 ) = (s ) x 00 (s )(pr c ) > 0, (5) (rt s ) 2 whereas s 0 (pr ) = 0 for pr < c + (+ rt 2 x 0 (0) ). We also notce that for pr > c + (+ rt 2 x 0 (0) ), s ncreases concavely wth pr. The value of s (p ) obtaned from the soluton of the equaton x 0 (s )(pr c ) = (rt s ) 2 f pr c + ( + rt 2 x 0 (0)), can be fed nto Eq. (5) to compute the prce vector. The system of equatons that result after substtuton s non-lnear n vector pr could have multple solutons,.e., multple Nash equlbra.

9 20 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) 3 24 Inferences from senstvty analyss: Games, 2, 3 gve us non-ntutve nsghts to the prce QoS changes by ndvdual CPs. We observe that the obvous ntutons of equlbrum prce decrease of competng CPs wth ncreasng QoS levels vce-versa do not hold under all stuatons senstvty analyss provde the condtons under whch the counter-result holds. Thus, the ntrcate nature of non-cooperatve strategy selecton by ndvdual CPs the nterdependences of ndvdual strateges on the cloud market make cloud economcs problems nterestng. Convergence to Nash equlbra: Snce multple Nash equlbra mght exst for the prce vectors for the smultaneous prce QoS game, the tatonnement scheme [9,2] can be used to prove convergence. Ths scheme s an teratve procedure that numercally verfes whether multple prce equlbra exst, unqueness s guaranteed f only f the procedure converges to the same lmt when ntal values are set at pr mn or pr max. Once the equlbrum prce vectors are determned, the equlbrum servce levels are easly computed. If multple equlbra exst the cloud provders select the prce equlbra that s component-wse the largest. Regardng the case when CP prce vector s gven, we have the followng corollary from the result of Theorem 2, whch leads us to equlbrum results of Game 3, a specal case of Game 2. Corollary 2. Gven any CP prce vector, pr f, the Nash equlbrum s(pr f ) s the domnant soluton n the QoS level game between CPs,.e., a CP s equlbrum QoS level s ndependent of any of ts compettors cost or dem characterstcs prces. When s (pr f )>0, the equlbrum QoS level s ncreasng concave n pr f, wth s0 (pr f ) = x 0 (s ) x 00 (s )(pr f c ) 2. (rt s ) 3 We observe that Game 3 beng a specal case of Game 2 entals a unque Nash equlbrum, whereas Game 2 entals multple Nash equlbra. 5. Optmzaton framework for capacty provsonng In ths secton, we develop optmzaton models for optmally provsonng capacty n both, sngle-ter as well as multter cloud networks. As mentoned n prevous sectons, the term capacty has a queueng-theoretc noton to t s the servce rate of a queueng system processng user requests,.e., t s the number of user requests processed per unt of tme. The capacty measure can be translated to allocatng hardware other system resources optmally so as to satsfy user QoS dems. In the followng subsectons, we frst deal wth the capacty analyss n sngle ter clouds, whch s followed by the analyss n mult-ter cloud networks. 5.. Sngle-ter case We model each CP as an M/M/ queueng system wth frst-come, frst-serve (FCFS) schedulng, where end-user requests arrve as a Posson process wth mean rate, gets servced at a rate µ. We adopt an M/M/ queueng system because of three reasons: () queueng theory has been tradtonally used n request arrval servce problems, (2) for our problem, assumng an M/M/ queueng system ensures tractable analyses procedures that entals dervng nce closed form expressons helps underst system nsghts n a non-complex manner, wthout sacrfcng a great deal n capturng the real dynamcs of the actual arrval departure process, (3) the Markovan nature of the servce process helps us generalze expected steady state analyss percentle analyss together. We assume that each CP adopts the FCFS schedulng polcy because they serve a sngle class of end-users wth the same QoS level guarantees. The metrc for end-user satsfacton n queueng systems s response/watng tme. The response tme rt may be defned, ether n terms of the expected steady state response tme,.e., rt = E(RT ), or n terms of -percentle performance, rt ( ), where 0 < <. Thus, n terms of -percentle performance, 0 P(RT < rt ( )) =. Accordng to the theory of M/M/ queues, we have the followng stard results [7]. rt = µ, rt ( ) = ln( ), µ ( ) µ = +, rt µ ( ) = + ln( ) rt ( ). (6) (7) (8) (9) 0 As an example, n cloud networks we often assocate provsonng power accordng the 95th percentle use. Lkewse, we could also provson servce capacty by accountng for percentle response tme guarantees.

10 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) Eqs. (20) (22) follow from the fact that for M/M/ queues, the followng result holds, P(RT < rt ( )) = = e (µ rt ( )). (20) The nverse of rt (rt ( )) s s (s ( )), whch s the advertsed QoS level guarantee of CP to ts end-users. Thus, we observe from Eqs. (2) (22) that the queueng servce rate (capacty) s lnear n s (s ( )). Snce C s proportonal to µ (µ ( )), we nfer that C s lnear n s (q ( )). Our am n ths paper s to fnd the optmal µ(µ ( )) for each CP such that ts advertsed QoS level guarantees to ts end-users are satsfed, wthout wastng any resources. Assumng that t takes a cost of for CP to provson a sngle unt of servce capacty, we have the followng optmzaton problems consderng the expected value percentle value of response tme respectvely. subect to subect to mn µ µ apple rt 8 mn µ ( ) log( ) µ ( ) 5.2. Mult-ter case apple rt ( ) 8. In order to model the mult-ter case, we model a gven cloud network for CP as a network of queues. Each queue n the network acts as an M/M/ queue servng end-user requests n an FCFS manner. We assume that the queueng network s an open Jackson network [7]. We also assume the queueng network for any CP s dstnct from other CP queung networks,.e., for CP, there s no queue n ts network that serves any other CP, 8 6=. Each queue s representatve of a ter n a cloud network s represented as a vertex/node n the open Jackson network. The departure process of one ter/level s an arrval process for the next ter. We defne the followng notatons n relaton to our analyss of queueng networks for CP V - set of n vertces n an open Jackson network for CP. - fracton of end user requests that start servcng at node. - probablty that a user request moves to node k after gettng servce from node. p k P - matrx of p k values s sub-stochastc n nature,.e., Lt n!(p ) n = 0 µ - servce rate of node V - capacty cost per unt of servce rate at node. - vector of aggregate arrval rates for CP. The vector of arrval rates for each CP s expressed as recursve expresson of the form = + (P ) T. Solvng the above equaton, we get =, where the vector = (I (P ) T ). Accordng to queueng theory results regardng networks of queues, we get the followng for expressons for each CP (for the expected value case of response tme) (2) (22) E[requests at node ] = µ (23) E[total number system requests] = V µ. (24) Due to the Markovan nature of the servce process, the case for general percentles s exactly smlar to the case for expected steady state analyss. The expressons reman nearly the same apart from a constant factor multplcaton.

11 22 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) 3 24 By Lttle s law, we have s =. µ V (25) We now prove through the followng theorem that even n mult-ter cloud networks, C s lnear n s, for each CP. Ths fact regardng lnearty s mportant when t comes to the case of analyzng prce QoS games. Theorem 3. The capacty provsonng cost, C, for each cloud provder n a mult-ter cloud network s lnear n ther user arrval rate the advertsed QoS level guarantee. Proof. Each cloud provder s wllng to mnmze ther capacty costs. Thus t selects µ = (µ : V ) such that t s the soluton of the followng constraned optmzaton problem mn subect to V µ µ V apple s. Applyng Karush Kuhn Tucker (KKT) condtons [25] for optmalty, we have where = µ (opt) (µ (opt), V, )2 s the Lagrange multpler. From the prevous equaton we get v = p u t, V. (26) (27) The mnmum cost of CP evaluates to P V µ (opt), whch s of the form A + A 2 s, where A = V (28) 0 A2 q V A 2. (29) Thus, the capacty provsonng cost per CP n a mult-ter cloud network s lnear n ther user arrval rate the advertsed QoS level guarantee. We emphasze that the theorem holds (due to the Markovan nature of the servce tmes) when we consder the response-tme as a percentle parameter, rather than an expected value. Optmzaton Problems: We have the followng two optmzaton problems for mult-ter networks consderng the expected value percentle value of response tme respectvely. mn subect to subect to V µ µ V mn µ( ) V V log apple s µ ( ) apple s ( ). The optmzaton problems for the sngle-ter mult-ter cases provde a framework va whch resources can be provsoned n the cloud n a manner so as to mnmze over-provsonng n a dynamc manner.

12 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) Concluson future work In the frst part of the paper, we developed nter-organzatonal economc models for prcng cloud network servces when several cloud provders co-exst n a market, servcng a sngle applcaton type. We devsed analyzed three prce QoS game-theoretc models relevant to cloud networks. We proved that a unque pure strategy Nash equlbrum (NE) exsts n two of our three QoS-drven prcng models. In addton, we also showed that player dynamcs converge to NE s converge;.e., there s a practcally mplementable algorthm for each model that computes the NE/s for the correspondng model. Thus, even f no unque Nash equlbrum exsts n some of the models, we are guaranteed to fnd the largest equlbra (preferred by the CPs) through our algorthm. Regardng convergence to Nash equlbra, t s true that t could take a long tme for convergence of Nash equlbra (computng NE s PPAD Complete [8]), however n 95% of the cases n practcal economc markets, NE s acheved n a decent amount of tme. Our prce QoS models can drve optmal resource provsonng n cloud networks. The NE prce QoS levels for each cloud provder drve optmal end-user dem n a gven tme perod w.r.t. maxmzng ndvdual CP profts under competton. Servcng end-user dems requres provsonng capacty. Once the optmal values are computed, the power of vrtualzaton n cloud networks makes t possble to execute dynamc resource provsonng n a fast effcent manner n multple tme perods. In ths regard, n the second part of the paper, we developed an optmzaton framework for sngletered mult-tered cloud networks to compute the optmal provsoned capacty once the equlbrum prce QoS levels for each CP have been determned. As part of future work, we plan to extend our analyss to the case where cloud provders are n smultaneous competton wth other CPs on multple applcaton types. Appendx Proof of Theorem. For a gven servce level vector s, each CP reserves a capacty of rt =!. Consder the game G wth rt s proft/utlty functons for each CP represented as P = x (s ) y p (s ) + ( p )(pr c ) W, (30) 6= 6= where Snce W = rt P =, the functon P s supermodular. 2 The strategy set of each CP les nsde a closed nterval s bounded,.e., the strategy set s [pr mn, pr max ], whch s a compact set. Thus, the prcng game between CPs s a supermodular game possesses a Nash equlbrum [3]. Snce y > P 6=, =,...,n (by Eq. (7)), equlbrum s unque. Rewrtng Eq. () usng Eq. (6), we get P = y (pr c ) 2 rt 2 P > 2 2 6= thus = y (pr c ). Substtutng n Eq. (9), we get Proof of Corollary. Snce the nverse of matrx M,.e., M exsts s greater than or equal to 0 [4], from pr = M (x(s) + z) (Eq. ()), we have pr s ncreasng n {c, =, 2,...,n}. Agan, from Lemma 2 n [4], we have y (M ) ) 0.5 apple <, where s the degree of postve externalty 3 faced by CP from other CP (prce, QoS) parameters, t ncreases wth the coeffcents. Ths leads @ = y (M ) = > 0. Therefore, we show n another dfferent way that pr s ncreasng n {c,, =, 2,...,n}. Snce M exsts s greater than or equal to 0, we @ = ) = ) = y ( )<0, from whch we conclude that s decreasng n {c,, =, 2,...,n}. Part (b) of the corollary drectly follows from the fact that the nverse of matrx M,.e., M exsts, s greater than or equal to 0, every entry of M s ncreasng n coeffcents. Proof of Theorem 2. To prove our theorem, we ust need to show that the proft functon P s ontly concave n (pr, s ). Then by the Nash Debreu theorem [5], we could nfer the exstence of a Nash equlbra. We know the followng results for = y (pr c ) + (3) 2 A functon f : Rn! R s supermodular f t has the followng ncreasng dfference property,.e., f (m, m ) f (m 2, m ), ncreases n m for all m > m 2 n (pr, pr ). The readers are referred to [6] for more detals on supermodularty. 3 A postve externalty s an external beneft on a user not drectly nvolved n a transacton. In our case, a transacton refers to a CP settng ts prce QoS parameters.

13 24 R. Pal, P. Hu / Theoretcal Computer Scence 496 (203) = x 0 (s )(pr c ) (rt s ) 2. 2 P Thus, = 2 < P = x 2 Hessan as 2y (x 00 (s )(pr c ) followng condton holds: 2 (x 0 (s )) 2, rt apple mn s 3 (s )(pr c ) s 2 (rt s ) 3 < = x 0 (s ). We determne the determnant of the (rt s ) 2 0 (the suffcent condton for P to be ontly concave n (pr, s )), f the s 4y (x 0 (s )) 2 = 3 4y (x 0, (33) (0))2 where the last equalty follows from the fact that x 0 > 0 x 0 s decreasng. Now snce pr = pr (s ), by Theorem t s n the closed bounded nterval [pr mn, pr max ] must therefore satsfy Eq. (5). Agan from Eq. (3), as s tends to rt, whch leads us to the concluson that s (pr ) s the unque root of x 0 (s )(pr c ) = (rt s ) 2 f pr c + ( + ) rt 2 x 0 or s (pr (0) ) = 0 otherwse. Proof of Corollary 2. Substtutng pr max = pr mn = pr f nto Theorem 2 leads us to the fact that s(pr f ) s a Nash equlbrum of the QoS level competton game amongst CPs that t s also a unque a domnant soluton, snce s(pr f ) s a functon of pr, c, only. (Followng from the fact that s (pr ) s the unque root of x 0 (s )(pr c ) = (rt s ) f pr 2 c + ( + ) rt 2 x 0 (0) or s (pr ) = 0 otherwse.) References [] M. Armbrust, A. Fox, R. Grffth, A.D. Joseph, R.H. Katz, A. Konwnsk, G. Lee, D.A. Patterson, A. Rabkn, I. Stoca, M. Zahara, Above the clouds: a Berkeley vew of cloud computng, Techncal Report, EECS, U. C. Berkeley, [2] S.C.M. Lee, J.C.S. Lu, On the nteracton competton among Internet servce provders, IEEE Journal on Selected Areas n Communcatons 26 (2008). [3] S. Shakkota, R. Srkant, Economcs of network prcng wth multple ISPs, IEEE/ACM Transactons on Networkng 4 (2006). [4] P. He, M. Chang, R. Calderbank, S. Rangan, Network prcng rate allocaton wth content-provder partcpaton, n: IEEE INFOCOM, 200. [5] L. Jang, S. Parekh, J. Walr, Tme-dependent network prcng bwdth tradng, n: IEEE BoD, [6] J.K. Macke-Mason, H.R. Varan, Prcng congestble network resources, IEEE Journal on Selected Areas n Communcatons 3 (995). [7] D. Songhurst, F. Kelly, Chargng schemes for multservce networks, n: 5th Internatonal Teletra?c Congress, 997. [8] P. Dube, R. Jan, C. Touat, An analyss of prcng competton for queued servces wth multple provders, n: ITA Workshop, [9] H.R. Varan, Mcroeconomc Analyss, Norton, 992. [0] M.E. Wetzsten, Mcroeconomc Theory: Concepts Connectons, South Western, [] W. Rudn, Prncples of Mathematcal Analyss, Mc.Graw Hll, 976. [2] K. Arrow, Hbook of Mathematcal Economcs, North Holl, 98. [3]. Vves, Nash equlbrum strategc complementartes, Journal of Mathematcal Economcs 9 (990). [4] F. Bernsten, A. Federgruen, Compartve statcs, strategc complements, substtutes n olgopoles, Journal of Mathematcal Economcs 40 (2004). [5] D. Fudenberg, J. Trole, Game Theory, MIT Press, 99. [6] D.M. Topks, Supermodularty Complementarty, Prnceton Unversty. [7] D. Bertsekas, R. Gallager, Data Networks, Prentce Hall Inc, 988. [8] C. Daskalaks, P.W. Goldberg, C.H. Papadmtrou, The complexty of computng a Nash equlbrum, SIAM Journal of Computng 39 () (2009). [9] D. Gmach, J. Rola, L. Cherkasova, A. Kemper, Capacty management dem predcton for next generaton data centers, n: IEEE Internatonal Conference on Web Servces, [20] T. Wood, L. Cherkasova, K. Ozonat, P. Shenoy, Proflng modelng resource usage of vrtualzed applcatons, n: ACM Internatonal Conference on Mddleware, [2] D. Kusc, N. Kasamy, Rsk-aware lmted lookahead control for dynamc resource provsonng n enterprse computng systems, n: IEEE ICAC, [22] P. Padala, K.G. Shn,. Zhu, M. Uysal, Z. Wang, S. Snghal, A. Merchant, K. Salem, Adaptve control of vrtualzed resources n utlty computng envronments, n: ACM SIGOPS, [23]. Meng, C. Isc, J. Kephart, L. Zhang, E. Boulett, Effcent resource provsonng n compute clouds va VM multplexng, n: ACM ICAC, 200. [24] J. Deun, G. Perre, C-H. Ch, Autonomous resource provsonng for mult-servce web applcatons, n: ACM WWW, 200. [25] S. Boyd, L. Verberghe, Convex Optmzaton, Cambrdge Unversty Press, 2005.