An Analytical Model for Multi-tier Internet Services and Its Applications

Transcription

1 An Analytcal Model for Mult-ter Internet Servces and Its Alcatons Bhuvan Urgaonkar, Govann Pacfc, Prashant Shenoy, Mke Sretzer, and Asser Tantaw Det. of Comuter Scence, Servce Management Mddleware Det., Unversty of Massachusetts, IBM T. J. Watson Research Center, Amherst, MA 13 Hawthorne, NY ABSTRACT Snce many Internet alcatons emloy a mult-ter archtecture, n ths aer, we focus on the roblem of analytcally modelng the behavor of such alcatons. We resent a model based on a network of queues, where the queues reresent dfferent ters of the alcaton. Our model s suffcently general to cature () the behavor of ters wth sgnfcantly dfferent erformance characterstcs and () alcaton dosyncrases such as sesson-based workloads, concurrency lmts, and cachng at ntermedate ters. We valdate our model usng real mult-ter alcatons runnng on a Lnux server cluster. Our exerments ndcate that our model fathfully catures the erformance of these alcatons for a number of workloads and confguratons. For a varety of scenaros, ncludng those wth cachng at one of the alcaton ters, the average resonse tmes redcted by our model were wthn the 95% confdence ntervals of the observed average resonse tmes. Our exerments also demonstrate the utlty of the model for dynamc caacty rovsonng, erformance redcton, bottleneck dentfcaton, and sesson olcng. In one scenaro, where the request arrval rate ncreased from less than 15 to nearly 42 requests/mn, a dynamc rovsonng technque emloyng our model was able to mantan resonse tme targets by ncreasng the caacty of two of the alcaton ters by factors of 2 and 3.5, resectvely. Categores and Subject Descrtors C.4 [Performance of Systems]: Modelng Technques General Terms Measurement, Performance, Exermentaton Portons of ths research were done when Bhuvan Urgaonkar was a summer ntern at IBM T. J. Watson Research Center. Ths research was suorted n art by NSF grants CCR-99843, CNS , and EIA Permsson to make dgtal or hard coes of all or art of ths work for ersonal or classroom use s granted wthout fee rovded that coes are not made or dstrbuted for roft or commercal advantage and that coes bear ths notce and the full ctaton on the frst age. To coy otherwse, to reublsh, to ost on servers or to redstrbute to lsts, requres ror secfc ermsson and/or a fee. SIGMETRICS 5, June 6 1, 25, Banff, Alberta, Canada. Coyrght 25 ACM /5/6 $5.. Keywords Queung model, MVA algorthm, Internet alcaton 1. INTRODUCTION 1.1 Motvaton Internet alcatons such as onlne news, retal, and fnancal stes have become commonlace n recent years. Modern Internet alcatons are comlex software systems that emloy a mult-ter archtecture and are relcated or dstrbuted on a cluster of servers. Each ter rovdes a certan functonalty to ts recedng ter and makes use of the functonalty rovded by ts successor to carry out ts art of the overall request rocessng. For nstance, a tycal e-commerce alcaton conssts of three ters a front-end Web ter that s resonsble for HTTP rocessng, a mddle ter Java enterrse server that mlements core alcaton functonalty, and a backend database that stores roduct catalogs and user orders. In ths examle, ncomng requests undergo HTTP rocessng, rocessng by Java alcaton server, and trgger queres or transactons at the database. Ths aer focuses on analytcally modelng the behavor of multter Internet alcatons. Such a model s mortant for the followng reasons: () caacty rovsonng, whch enables a server farm to determne how much caacty to allocate to an alcaton n order for t to servce ts eak workload; () erformance redcton, whch enables the resonse tme of the alcaton to be determned for a gven workload and a gven hardware and software confguraton, () alcaton confguraton, whch enables varous confguraton arameters of the alcaton to be determned for a certan erformance goal, (v) bottleneck dentfcaton and tunng, whch enables system bottlenecks to be dentfed for uroses of tunng, and (v) request olcng, whch enables the alcaton to turn away excess requests durng transent overloads. Modelng of sngle-ter alcatons such as vanlla Web servers (e.g., Aache) s well studed [4, 12, 17]. In contrast, modelng of mult-ter alcatons s less well studed, even though ths flexble archtecture s wdely used for constructng Internet alcatons and servces. Extendng sngle-ter models to mult-ter scenaros s non-trval due to the followng reasons. Frst, varous alcaton ters such as Web, Java, and database servers have vastly dfferent erformance characterstcs and collectvely modelng ther behavor s a dffcult task. Further, n a mult-ter alcaton, () there may be concurrency lmts at one or more ters, and () cachng may be emloyed at ntermedate ters all of whch comlcate the erformance modelng. Fnally, modern Internet workloads are sesson-based, where each sesson comrses a sequence of re-

2 quests wth thnk-tmes n between. For nstance, a sesson at an onlne retaler comrses the sequence of user requests to browse the roduct catalog and to make a urchase. Sessons are stateful from the ersectve of the alcaton, an asect that must be ncororated nto the model. The desgn of an analytcal model that can cature the mact of these factors s the focus of ths aer. 1.2 Research Contrbutons Ths aer resents a model of a mult-ter Internet alcaton based on a network of queues, where the queues reresent dfferent ters of the alcaton. Our model can handle alcatons wth an arbtrary number of ters and those wth sgnfcantly dfferent erformance characterstcs. A key contrbuton of our work s that the comlex task of modelng a mult-ter alcaton s reduced to the modelng of request rocessng at ndvdual ters and the flow of requests across ters. Our model s nherently desgned to handle sesson-based workloads and can account for alcaton dosyncrases such as cachng effects and concurrency lmts at each ter. We valdate the model usng two oen-source mult-ter alcatons runnng on a Lnux-based server cluster. We demonstrate the ablty of our model to accurately cature the effects of a number of commonly used technques such as query cachng at the database ter and class-based servce dfferentaton. For a varety of scenaros, ncludng an onlne aucton alcaton emloyng query cachng at ts database ter, the average resonse tmes redcted by our model were wthn the 95% confdence ntervals of the observed average resonse tmes. We conduct a detaled exermental study usng our rototye to demonstrate the utlty of our model for the uroses of dynamc rovsonng, resonse tme redcton, alcaton confguraton, and request olcng. Our exerments demonstrate the ablty of our model to correctly dentfy bottlenecks n the system and the shftng of bottlenecks due to varatons n the Internet workload. In one scenaro, where the arrval rate to an alcaton ncreased from 15 to nearly 42 requests/mn, our model was able to contnue meetng resonse tme targets by successfully dentfyng the two bottleneck ters and ncreasng ther caacty by factors of 2 and 3.5, resectvely. The remander of ths aer s structured as follows. Secton 2 rovdes an overvew of mult-ter alcatons and related work. We descrbe our model n Sectons 3 and 4. Sectons 6 and 7 resent exermental valdaton of the model and an llustraton of ts alcatons resectvely. Fnally, Secton 8 resents our conclusons. 2. BACKGROUND AND RELATED WORK Ths secton rovdes an overvew of mult-ter alcatons and the underlyng server latform assumed n our work. We also dscuss related work n the area. 2.1 Internet Alcaton Archtecture Modern Internet alcatons are desgned usng multle ters (the terms Internet alcaton and servce are used nterchangeably n ths aer). A mult-ter archtecture rovdes a flexble, modular aroach for desgnng such alcatons. Each alcaton ter rovdes certan functonalty to ts recedng ter and uses the functonalty rovded by ts successor to carry out ts art of the overall request rocessng. The varous ters artcate n the rocessng of each ncomng request durng ts lfetme n the system. Deendng on the rocessng demand, a ter may be relcated usng clusterng technques. In such an event, a dsatcher s used at each relcated ter to dstrbute requests among the relcas for the urose of load balancng. Fgure 1 dects a three-ter alcaton where the frst two ters are relcated, whle the thrd one s not. Such an archtecture s commonly emloyed by e-commerce al- Sentry Polcng Dro sessons (f needed) Load Balancer Ter 1 dsatcher Ter 1 Indvdual server Ter 2 dsatcher Ter 2 Fgure 1: A three-ter alcaton. Ter 3 (non relcated) catons where a clustered Web server and a clustered Java alcaton server consttute the frst two ters, and the thrd ter conssts of a non-relcable database. 1 The workload of an Internet alcaton s assumed to be sessonbased, where a sesson conssts of a successon of requests ssued by a clent wth thnk tmes n between. If a sesson s stateful, whch s often the case, successve requests wll need to be servced by the same server at each ter, and the dsatcher wll need account for ths server state when redrectng requests. As shown n Fgure 1, each alcaton emloys a sentry that olces ncomng sessons to an alcaton s server ool ncomng sessons are subjected to admsson control at the sentry to ensure that the contracted erformance guarantees are met; excess sessons are turned away durng overloads. We assume that Internet alcatons tycally run on a server cluster that s commonly referred to as a data center. In ths work, we assume that each ter of an alcaton (or each relca of a ter) runs on a searate server. Ths s referred to as dedcated hostng, where each alcaton runs on a subset of the servers and a server s allocated to at most one alcaton ter at any gven tme. Unlke shared hostng where multle small alcatons share each server, dedcated hostng s used for runnng large clustered alcatons where server sharng s nfeasble due to the workload demand mosed on each ndvdual alcaton. Gven an Internet alcaton, we assume that t secfes ts desred erformance requrement n the form of a servce-level agreement (SLA). The SLA assumed n ths work s a bound on the average resonse tme that s accetable to the alcaton. For nstance, the alcaton SLA may secfy that the average resonse tme should not exceed one second regardless of the workload. 2.2 Request Processng n Mult-ter Alcatons Consder a mult-ter alcaton consstng of M ters denoted by T 1, T 2 through T M. In the smlest case, each request s rocessed exactly once by ter T and then forwarded to ter T +1 for further rocessng. Once the result s comuted by the fnal ter T M, t s sent back to T M 1, whch rocesses ths result and sends t to T M 2 and so on. Thus, the result s rocessed by each ter n the reverse order untl t reaches T 1, whch then sends t to the clent. Fgure 2 llustrates the stes nvolved n rocessng a bd request at a three-ter onlne aucton ste. The fgure shows how the request trckles downstream and how the result roagates ustream through the varous ters. More comlex rocessng at the ters s also ossble. In such scenaros, each request can vst a ter multle tmes. As an ex- 1 Tradtonally database servers have emloyed a shared-nothng archtecture that does not suort relcaton. However, certan new databases emloy a shared-everythng archtecture [13] that suorts clusterng and relcaton but wth certan constrants.

3 amle, consder a keyword search at an onlne suerstore, whch trggers a query on the musc catalog, a query on the book catalog and so on. These queres can be ssued to the database ter sequentally, where each query s ssued after the result of the revous query has been receved, or n arallel. Thus, n the general case, each request at ter T can trgger multle requests to ter T +1. In the sequental case, each of these requests s ssued to T +1 once the result of the revous request has fnshed. In the arallel case, all requests are ssued to T +1 at once. In both cases, all results are merged and then sent back to the ustream ter T 1. Clent 1. Clent 2. HTTP 3. J2EE 4. J2EE 5. J2EE 6. HTTP HTTP server HTTP (lace bd on some tem) J2EE (servlet nvokes EJB) Database (EJB ssues queres, database resonds) (EJB constructs resonse) HTTP (resonse sent to HTTP server) Clent (resonse sent to clent) 5 4 J2EE server 3 Database server 2.3 Related Work Modelng of sngle-ter Internet alcatons, of whch HTTP servers are the most common examle, has been studed extensvely. A queung model of a Web server servng statc content was roosed n [17]. The model emloys a network of four queues two modelng the Web server tself, and the other two modelng the Internet communcaton network. A queung model for erformance redcton of sngle-ter Web servers wth statc content was roosed n [4]. Ths aroach () exlctly models CPU, memory, and dsk bandwdth n the Web server, () utlzes knowledge of fle sze and oularty dstrbutons, and () relates average resonse tme to avalable resources. A GPS-based queung model of a sngle resource, such as the CPU, at a Web server was roosed n [3]. The model s arameterzed by onlne measurements and s used to determne the resource allocaton needed to meet desred average resonse tme targets. A G/G/1 queung model for relcated sngle-ter alcatons (e.g., clustered Web servers) has been roosed n [18]. The archtecture and rototye mlementaton of a erformance management system for cluster-based Web servces was roosed n [11]. The work emloys an M/M/1 queung model to comute resonses tmes of Web requests. A model of a Web server for the urose of erformance control usng classcal feedback control theory was studed n [1]; an mlementaton and evaluaton usng the Aache Web server was also resented n the work. A combnaton of a Markov chan model and a queung network model to cature the oeraton of a Web server was resented n [12] the former model reresents the software archtecture emloyed by the Web server (e.g. rocess-based versus thread-based) whle the latter comutes the Web server s throughut. Snce these efforts focus rmarly on sngle-ter Web servers, they are not drectly alcable to alcatons emloyng multle ters, or to comonents such as Java enterrse servers or database servers emloyed by mult-ter alcatons. Further, many of the above efforts assume statc Web content, whle mult-ter alcatons, by ther very nature, serve dynamc Web content. A few recent efforts have focused on the modelng of mult-ter alcatons. However, many of these efforts ether make smlfyng assumtons or are based on smle extensons of sngle-ter models. A number of aers have taken the aroach of modelng only the most constraned or the most bottlenecked ter of the alcaton. For nstance, [2] consders the roblem of rovsonng servers for only the Java alcaton ter; t uses an M/G/1/PS model for each server n ths ter. Smlarly, the Java alcaton ter of an e-commerce alcaton wth N servers s modeled as a G/G/N queung system n [14]. Other efforts have modeled the entre mult-ter alcaton usng a sngle queue an examle s [7], that uses a M/GI/1/PS model for an e-commerce alcaton. Whle these aroaches are useful for secfc scenaros, they have many lmtatons. For nstance, modelng only a sngle bottlenecked ter of a mult-ter alcaton wll fal to cature cachng effects at other ters. Such a model can not be used for caacty rovsonng of other ters. Fnally, as we show n our exerments, system Fgure 2: Request rocessng n an onlne aucton alcaton. bottlenecks can shft from one ter to another wth changes n workload characterstcs. Under these scenaros, there s no sngle ter that s the most constraned. In ths aer, we resent a model of a mult-ter alcaton that overcomes these drawbacks. Our model exlctly accounts for the resence of all ters and also catures alcaton artfacts such as sesson-based workloads, cachng effects, and concurrency lmts. 3. A MODEL FOR A MULTI-TIER INTERNET APPLICATION In ths secton, we resent a baselne queung model for a multter Internet alcaton, followed by enhancements to the model to cature certan alcaton dosyncrases. 3.1 The Basc Queung Model Consder an alcaton wth M ters denoted by T 1,, T M. To begn wth we assume that no ter s relcated each ter s assumed to run on exactly one server. In Secton 5 we descrbe a smle enhancement to our model to cature the resence of relcated ters. Modelng Multle Ters: We model the alcaton usng a network of of M queues, Q 1,, Q M (see Fgure 3). Each queue reresents an alcaton ter and the underlyng server that t runs on. We assume a rocessor sharng (PS) dsclne at each queue, snce t closely aroxmates the schedulng olces emloyed by most commodty oeratng systems (e.g., Lnux CPU tme-sharng). When a request arrves at ter T t trggers one or more requests at ts subsequent ter T +1; recall the examle of a keyword search that trggers multle queres at dfferent roduct catalogs. In our queung model, we can cature ths henomenon by allowng a request to make multle vsts to each of the queues durng ts overall executon. Ths s acheved by ntroducng a transton from each queue to ts redecessor, as shown n Fgure 3. A request, after some rocessng at queue Q, ether returns to Q 1 wth a certan robablty or roceeds to Q +1 wth robablty (1 ). The only excetons are the last ter queue Q M, where all requests return to the revous queue, and the frst queue Q 1, where a transton to the recedng queue denotes request comleton. As argued n Secton 3.2, our model can handle multle vsts to a ter regardless of whether they occur sequentally or n arallel. Observe that cachng effects are naturally catured by ths model. If cachng s emloyed at ter T, a cache ht causes the request to mmedately return to the revous queue Q 1 wthout trggerng any work n queues Q +1 or later. Thus, the mact of cache hts and msses can be ncororated by arorately determnng the transton robablty and the servce tme of a request at Q. Modelng Sessons: Recall from Secton 2 that Internet workloads are sesson-based. A sesson ssues one or more requests dur-

4 Z Z Z Sessons Q 1 Q Q 2 S 1 S S 2 M M 1 M Q M Ter 1 Ter 2 Ter M Fgure 3: Modelng a mult-ter alcaton usng a network of queues. ng ts lfetme, one after another, wth thnk tmes n between (we refer to ths duraton as the user thnk tme). Tycal sessons n an Internet alcaton may last several mnutes. Thus, our model needs to cature the relatvely long-lved nature of sessons as well as the resonse tmes of ndvdual requests wthn a sesson. We do so by augmentng our queung network wth a subsystem modelng the actve sessons of the alcaton. We model sessons usng an nfnte server queung system, Q, that feeds our network of queues and forms the closed-queung system shown n Fgure 3. The servers n Q cature the sesson-based nature of the workload as follows. Each actve sesson s assumed to occuy one server n Q. As shown n Fgure 3, a request ssued by a sesson emanates from a server n Q and enters the alcaton at Q 1. It then moves through the queues Q 1,, Q M, ossbly vstng some queues multle tmes (as catured by the transtons from each ter to ts recedng ter) and gettng rocessed at the vsted queues. Eventually, ts rocessng comletes, and t returns to a server n Q. The tme sent at ths server models the thnk tme of the user; the next request of the sesson s ssued subsequently. The nfnte server system also enables the model to cature the ndeendence of the user thnk tmes from the request servce tmes at the alcaton. Let S denote the servce tme of a request at Q (1 M). Also, denotes the robablty of a request makng a transton from Q to Q 1 (note that M = 1); 1 denotes the robablty of transton from Q 1 to Q. Fnally, let Z denote the servce tme at any server n Q (whch s essentally the user thnk tme). Our model requres these arameters as nuts n order to comute the average end-to-end resonse tme of a request. Our dscusson thus far has mlctly assumed that sessons never termnate. In ractce, the number of sessons beng servced wll vary as exstng sessons termnate and new sessons arrve. Our model can comute the mean resonse tme for a gven number of concurrent sessons N. Ths roerty can be used for admsson control at the alcaton sentry, as dscussed n Secton Dervng Resonse Tmes From the Model The Mean-Value Analyss (MVA) algorthm [15] for closed-queung networks can be used to comute the mean resonse tme exerenced by a request n our network of queues. The MVA algorthm s based on the followng key queung theory result: In roductform closed queung networks 2, when a request moves from queue 2 The term roduct-form ales to any queung network n whch the exresson for the equlbrum robablty has the form of P (n 1,, n M ) = 1 G(N) πm =1f (n ) where f (n 1) s some functon of the number of jobs at the th queue, G(N) s a normalzng constant. Product form solutons are known to exst for a broad Q to another queue Q j, t sees, at the tme of ts arrval at Q j, a system wth the same statstcs as a system wth one less customer. Consder a roduct-form closed-queung network wth N customers. Let Ā m(n) denote the average number of customers n queue Q m seen by an arrvng customer. Let L m(n) denote the average length of queue Q m n such a system. Then, the above result mles Ā m(n) = L m(n 1) (1) Gven ths result, the MVA algorthm teratvely comutes the average resonse tme of a request. The MVA algorthm uses Equaton 1 to ntroduce customers nto the queung network, one by one, and determnes the resultng average delays at varous queues at each ste. It termnates when all N customers have been ntroduced, and yelds the average resonse tme exerenced by N concurrent customers. Note that a sesson n our model corresonds to a customer n the result descrbed by Equaton 1. The MVA algorthm for an M-ter Internet alcaton servcng N sessons smultaneously s resented n Algorthm 1 and the assocated notaton s n Table 1. The algorthm uses the noton of a vst rato for each queue Q 1,, Q M. The vst rato V m for queue Q m (1 m M) s defned as the average number of vsts made by a request to Q m durng ts rocessng (that s, from when t emanates from Q and when t returns to t). Vst ratos are easy to comute from the transton robabltes 1,, M and rovde an alternate reresentaton of the queung network. The use of vst ratos n leu of transton robabltes enables the model to cature multle vsts to a ter regardless of whether they occur sequentally or n arallel the vst rato s only concerned wth the mean number of vsts made by a request to a queue and not when or n what order these vsts occur. Thus, gven the average servce tmes and vst ratos for the queues, the average thnk tme of a sesson, and the number of concurrent sessons, the algorthm comutes the average resonse tme R of a request. 3.3 Estmatng the Model Parameters In order to comute the resonse tme, the model requres several arameters as nuts. In ractce, these arameters can be estmated by montorng the alcaton as t servces ts workload. To do so, we assume that the underlyng oeratng system and alcaton software comonents (such as the Aache Web server) rovde montorng hooks to enable accurate estmaton of these arameters. Our exerence wth the Lnux-based mult-ter alcatons used n our exerments s that such functonalty s ether already avalable or can be mlemented at a modest cost. The rest of ths secton descrbes how the varous model arameters can be estmated n ractce. Estmatng vst ratos: The vst rato for any ter of a multter alcaton s the average number of tmes that ter s nvoked durng a request s lfetme. Let λ req denote the number of requests servced by the entre alcaton over a duraton t. Then the vst rato for ter T can be smly estmated as V λ λ req where λ s the number of requests servced by that ter n that duraton. By choosng a sutably large duraton t, a good estmate for V can be obtaned. We note that the vst ratos are easy to estmate n an onlne fashon. The number of requests servced class of networks, ncludng ones where the schedulng dsclne at each queue s rocessor sharng (PS).

5 nut outut ntalzaton: R = D = Z; L = ; : N, S m, V m, 1 m M; Z : R m (avg. delay at Q m), R (avg. res. tme) for m = 1 to M do L m = ; D m = V m Sm /* servce demand at each queue */; end /* ntroduce N customers, one by one */ for n = 1 to N do for m = 1 to M do R m = D m(1 + L m) /* avg. delay at each que.*/; end ( ) n τ = R + M R /* throughut */; m=1 m for m = 1 to M do L m = τ R m /* udate queue lengths (lttle s law) */; end L = τ R ; end R = m=m m=1 R m /* resonse tme */; Algorthm 1: Mean-value analyss algorthm for an M-ter alcaton. by the alcaton λ req can be montored at the alcaton sentry. For each ter, the number of servced requests can be determned by real-tme rocessng of the ter logs. In the database ter, for nstance, the number of queres and transactons rocessed over a duraton t can be determned by rocessng the database log usng a scrt. Estmatng servce tmes: Alcaton comonents such as Web, Java, and database servers all suort extensve loggng facltes and can log a varety of useful nformaton about each servced request. In artcular, these comonents can log the resdence tme of ndvdual requests as observed at that ter the resdence tme ncludes the tme sent by the request at ths ter and all the subsequent ters that rocessed ths request. Ths loggng faclty can be used to estmate er-ter servce tmes. Let X denote the average er-request resdence tme at ter T. We start by estmatng the mean servce tme at the last ter. Snce ths ter does not nvoke servces from any other ters, the request executon tme at ths ter under lghtly loaded condtons s an excellent estmate of the servce tme. Thus, we have, S M X M Let S, X, and n be random varables denotng the servce tme of a request at a ter T, resdence tme of a request at ter T, and the number of tmes T requests servce from T +1 as art of the overall request rocessng, resectvely. Then, under lghtly loaded condtons, S = X n X +1, 1 < M. Takng averages on both sdes, we get, S = X E [n X +1] Snce n and X +1 are ndeendent, ths gves us, S = X n X +1 = X ( V+1 V ) X +1 Symbol Meanng M Number of alcaton ters N Number of sessons Q m Queue reresentng ter T m (1 m M) Q Inf. server system to cature sessons Z User thnk tme S m Avg. er-request servce tme at Q m L m Avg. length of Q m τ Throughut R m Avg. er-request delay at Q m R Avg. er-request resonse tme D m Avg. er-request servce demand at Q m V m Vst rato for Q m Ā m Avg. num. customers n Q m seen by an arrvng customer Table 1: Notaton used n descrbng the MVA algorthm. Thus, the servce tmes at ters T 1,, T M 1 can be estmated. Estmatng thnk tmes: The average user thnk tme, Z, can be obtaned by recordng the arrval and fnsh tmes of ndvdual requests at the sentry. Z s estmated as the average tme elased between when a request fnshes and when the next request (belongng to the same sesson) arrves at the sentry. By usng a suffcent number of observatons, we can obtan a good estmate of Z. Increased Servce Tmes Durng Overloads: Our estmaton of the ter-secfc servce tmes assumed lghtly loaded condtons. As the load on a ter grows, software overheads such as watng on locks, vrtual memory agng, and context swtch overheads, that are not catured by our model, can become sgnfcant comonents of the request rocessng tme. Incororatng the mact of ncreased context swtchng overhead or contenton for memory or locks nto our model s nontrval. Rather than exlctly modelng these effects, we mlctly account for ther mact by assocatng ncreased servce tmes wth requests under heavy loads. We use the Utlzaton Law [1] for a queung system whch states that S = ρ/τ, where ρ and τ are the queue utlzaton and throughut, resectvely 3. Consequently, we can mrove our estmate of the average servce tme at ter T as ( S = max S, ρ τ where ρ s the utlzaton of the busest resource (e.g. CPU, dsk, or network nterface) and τ s the ter throughut. Snce all modern oeratng systems suort facltes for montorng system erformance (e.g., the sysstat ackage n Lnux [16]), the utlzatons of varous resources are easy to obtan onlne. Smlarly, the ter throughut τ can be determned at the dsatcher (or from logs) by countng the number of comleted requests n a duraton t. 4. HANDLING CONCURRENCY LIMITS AT TIERS The software comonents of an Internet alcaton have lmts on the amount of concurrency they can handle. For nstance, the Aache Web server uses a confgurable arameter to lmt the number of concurrent threads or rocesses that are sawned to servce requests. Ths lmt revents the resdent memory sze of Aache 3 It should be noted that ρ and τ are measured values of utlzaton and throughut resectvely, and not values obtaned usng the MVA algorthm. )

6 Avg. res. tme (msec) Observed Basc Model Num. smult. sessons (a) Baselne model Avg. res. tme (msec) Observed Enhanced Model Num. smult. sessons (b) Enhanced model dro S 1 S S 2 M dro dro Q Q Q M M M Fgure 4: Resonse tme of Rubs wth 95% confdence ntervals. A concurrency lmt of 15 for Aache and 75 for the Java servlet ter s used. Fgure (a) dects the devaton of the baselne model from observed behavor when concurrency lmt s reached. Fgure (b) dects the ablty of the enhanced model to cature ths effect. from exceedng the avalable RAM and revents thrashng. Connectons are turned away when ths lmt s reached. Other ters mose smlar lmts. Ths secton rooses an enhancement to our baselne model to cature the effect of such lmts. Our baselne model assumes that each ter can servce an unbounded number of smultaneous requests and fals to cature the behavor of the alcaton when the concurrency lmt s reached at any ter. Ths s dected n Fgure 4(a), whch shows the resonse tme of a three-ter alcaton called Rubs that s confgured wth a concurrency lmt of 15 for the Aache Web server and a lmt of 75 for the mddle Java ter (detals of the alcaton aear n Secton 6.1). As shown, the resonse tmes redcted by the model match the observed resonse tmes untl the concurrency lmt s reached. Beyond ths ont, the model contnues to assume an ncreasng number of smultaneous requests beng servced and redcts an ncrease n resonse tme, whle the actual resonse tme of successful requests shows a flat trend due to an ncreasng number of droed requests. In general, when the concurrency lmt s reached at ter T, one of two actons are ossble: (1) the ter can slently dro addtonal requests and rely uon a tmeout mechansm n ter T 1 to detect these dros, or (2) the ter can exlctly notfy ter T 1 of ts nablty to serve the request (by returnng an error message). In ether case, ter T 1 may ressue the request some number of tmes before abandonng ts attemts. It wll then ether dro the request or exlctly notfy ts recedng ter. Fnally, ter T 1 can notfy the clent of the falure. Rather than dstngushng these ossbltes, we emloy a general aroach for caturng these effects. Let K denote the concurrency lmt at Q. To cature requests that are droed at ter T when ts concurrency lmt s reached, we add addtonal transtons, one for each queue reresentng a ter, to the basc model that we resented n Fgure 3. At the entrance of Q, we add a transton nto an nfnte server queung subsystem Q dro. Let dro denote the robablty of a request transtng from Q 1 to Q dro as shown n Fgure 5. Q dro has a mean servce tme of X dro. Ths enhancement allows us to dstngush between the rocessng of requests that get droed due to concurrency lmts and those that are rocessed successfully. Requests that are rocessed successfully are modeled exactly as n the basc model. Requests that are droed at ter T exerence some delay n the subsystem Q dro before returnng to Q ths models the delay between when a request s droed at ter T and when ths nformaton gets roagated to the clent that ntated the request. Q Q dro 1 Q dro 2 Sessons Ter 1 Ter 2 Ter M dro Q M Fgure 5: Mult-ter alcaton model enhanced to handle concurrency lmts. Lke n the baselne model, we can use the MVA algorthm to comute the resonse tme of a request. The algorthm comutes the fracton of requests that fnsh successfully and those that encounter falures, as well as the delays exerenced by both tyes of requests. To do so, we need to estmate the addtonal arameters that we have added to our basc model, namely, dro for each ter T. Estmatng dro the followng two stes. : Our aroach to estmate dro and X dro conssts of Ste 1 : Estmate throughut of the queung network f there were no concurrency lmts: Solve the queung network shown n Fgure 5 usng the MVA algorthm usng dro = (.e., assumng that the queues have no concurrency lmts). Let λ denote the throughut comuted by the MVA algorthm n ths ste. Ste 2 : Estmate dro : Treat Q as an oen, fnte-buffer M/M/1/K queue wth arrval rate λv (usng the λ comuted n Ste 1). Estmate dro as the robablty of buffer overflow n ths M/M/1/K queue [8]. Estmatng X dro : An estmate of X dro s alcaton-secfc and deends on the manner n whch nformaton about droed requests s conveyed to the clent, and how the clent resonds to t. In our current model we make the smlfyng assumton that uon detectng a faled request, the clent ressues the request. Ths s catured by the transtons from Q dro back to Q n Fgure 5. Our aroach for estmatng X dro s to subject the alcaton to an offlne workload that causes the lmt to be exceeded only at ter T (ths can be acheved by settng a low concurrency lmt at that ter and suffcently hgh lmts at all the other ters), and then record the resonse tmes of the requests that do not fnsh successfully. X dro s then estmated as the dfference between the average resonse tme of these unsuccessful requests and the sum of the servce tmes at ters T 1,, T 1 multled by the resectve vst ratos. In Fgure 4(b) we lot the resonse tmes for Rubs as redcted by our enhanced model. We fnd that ths enhancement enables us to cature the behavor of the Rubs alcaton even when ts concurrency lmt s reached.

7 5. OTHER ENHANCEMENTS AND SALIENT FEATURES Our closed queung model has several desrable features and can also be enhanced n other ways. Relcaton and load mbalances: Recall that our baselne model assumes a sngle server (queue) er ter and consequently does not suort the noton of relcaton at a ter. We now enhance our model to handle ths scenaro. Due to lack of sace, we resent the detals n [19]. Let r denote the number of relcas at ter T. Our aroach to cature relcaton at ter T s to relace the sngle queue Q wth r queues, Q 1,, Q r, one for each relca. A request n any queue can now make a transton to any of the r 1 queues of the revous ter or to any of the r +1 queues of the next ter. In general, whenever a ter s relcated, a dsatcher s necessary to dstrbute requests to relcas. The dsatcher determnes whch request to forward to whch relca and drectly nfluences the transtons made by a request. The dsatcher s also resonsble for balancng load across relcas. We make the smlfyng assumton of erfect load balancng. In a erfectly load balanced system, each relca rocesses 1 r fracton of the total workload of that ter. Ths mles that the vst ratos of the varous relcas at ter T can be chosen as V j = V /r In general, however, load mbalances may arse due to factors lke an affnty of sessons for artcular relcas. We are currently refnng our enhancement to take such effects nto account. Handlng Multle Sesson Classes: Internet alcatons tycally classfy ncomng sessons nto multle classes. To llustrate, an onlne brokerage Web ste may defne three classes and may ma fnancal transactons to the Gold class, customer requests such as balance nqures to the Slver class, and casual browsng requests from non-customers to the Bronze class. Tycally such classfcaton hels the alcaton sentry to referentally admt requests from more mortant classes durng overloads and dro requests from less mortant classes. We can extend our baselne model to account for the resence of dfferent sesson classes and to comute the resonse tme of requests wthn each class. Consder an Internet alcaton wth L sesson classes: C 1, C 2,..., C L. Assume that the sentry mlements a classfcaton algorthm to ma each ncomng sesson to one of these classes. We can use a straghtforward extenson of the MVA algorthm to deal wth multle sesson classes. We note that ths algorthm requres the vst ratos, servce tmes, and thnk tme to be measured on a er-class bass. Gven a L-tule (N 1,, N L) of sessons belongng to the L classes that are smultaneously servced by the alcaton, the algorthm can comute the average delays ncurred at each queue and the end-to-end resonse tme on a er-class bass. In Secton 7.2 we dscuss how ths algorthm can be used to flexbly mlement sesson olcng olces n an Internet alcaton. Our model currently does not handle the resence of multle sesson classes when concurrency lmts exst at the ters of an alcaton. Dealng wth multle sesson classes n the resence of concurrency lmts s art of ongong work. Smlcty: For an M-ter alcaton wth N concurrent sessons, the MVA algorthm has a tme comlexty of O(MN). The algorthm s smle to mlement, and as argued earler, the model arameters are easy to measure onlne. Generalty: Our model can handle an alcaton wth arbtrary number of ters. Further, when the schedulng dsclne s rocessor sharng (PS), the MVA algorthm works wthout makng any assumtons about the servce tme dstrbutons of the customers [1]. Ths feature s hghly desrable for two reasons: (1) t s reresentatve of schedulng olces n commodty oeratng systems (e.g., Lnux s CPU tme-sharng), and (2) t mles that our model s suffcently general to handle workloads wth an arbtrary servce tme requrements. 4 Whle our model s able to cature a number of alcaton dosyncrases, certan scenaros are not exlctly catured. Multle resources: We model each server occued by a ter usng a sngle queue. In realty, the server contans varous resources such as the CPU, dsk, memory, and the network nterface. Our model currently does not cature the utlzaton of varous server resources by a request at a ter. An enhancement to the model where varous resources wthn a server are modeled as a network of queues s the subject of future work. Resources held smultaneously at multle ters: Our model essentally catures the assage of a request through the ters of an alcaton as a juxtaoston of erods, durng each of whch the request utlzes the resources at exactly one ter. Although ths s a reasonable assumton for a large class of Internet alcatons, t does not aly to certan Internet alcatons such as streamng vdeo servers. A vdeo server that s constructed as a elne of rocessng modules wll have all of ts modules or ters actve as t contnuously rocesses and streams a vdeo to a clent. Our model does not aly to such alcatons. 6. MODEL VALIDATION In ths secton we resent our exermental setu followed by our exermental valdaton of the model. 6.1 Exermental Setu Alcatons: We use two oen-source mult-ter alcatons n our exermental study. Rubs mlements the core functonalty of an ebay lke aucton ste: sellng, browsng, and bddng. It mlements three tyes of user sessons, has nne tables n the database and defnes 26 nteractons that can be accessed from the clents Web browsers. Rubbos s a bulletn-board alcaton modeled after an onlne news forum lke Slashdot. Users have two dfferent levels of access: regular user and moderator. The man tables n the database are the users, stores, comments, and submssons tables. Rubbos rovdes 24 Web nteractons. Both alcatons were develoed by the DynaServer grou at Rce Unversty [5]. Each alcaton contans a Java-based clent that generates a sessonorented workload. We modfed these clents to generate the workloads and take the measurements needed by our exerments. We chose an average duraton of 5 mn for the sessons of both Rubs 4 The alcablty of the MVA algorthm s more restrcted wth some other schedulng dsclnes. E.g., n the resence of a FIFO schedulng dsclne at a queue, the servce tme at a queue needs to be exonentally dstrbuted for the MVA algorthm to be alcable.

8 Avg. res. tme (msec) Observed Model Num. smult. sessons Avg. resdence tme (msec) Obs. at Aache Obs. at Tomcat Model at Aache Model at Tomcat Num. smult. sessons Avg. CPU usage (%) Aache Tomcat Mysql Num. smult. sessons (a) Resonse tme (b) Resdence tmes (c) CPU utlzatons Fgure 7: Rubs based on Java servlets: bottleneck at CPU of database ter. The concurrency lmts for the Aache Web server and the Java servlets contaner were set to be 15 and 75, resectvely. Avg. resdence tme (msec) Aache, Obs. Tomcat, Obs. Aache, Basc Tomcat, Basc Aache, Enh. Tomcat, Enh Num. smult. sessons (a) Resdence tmes Avg. CPU usage (%) Aache Tomcat Mysql Num. smult. sessons (b) CPU utlzatons Fgure 6: Rubs based on Java servlets: bottleneck at CPU of mddle ter. The concurrency lmts for the Aache Web server and the Java servlets contaner were set to be 15 and 75, resectvely. and Rubbos. For both alcatons, the thnk tme was chosen from an exonental dstrbuton wth a mean of 1 sec. We used 3-ter versons of these alcatons. The front ter was based on Aache Web server. We exermented wth two mlementatons of the mddle ter for Rubs () based on Java servlets, and () based on Sun s J2EE Enterrse Java Beans (EJBs). The mddle ter for Rubbos was based on Java servlets. We emloyed Tomcat as the servlets contaner and JBoss as the EJB contaner. We used Kernel TCP Vrtual Server (ktcvs) verson..14 [9] to mlement the alcaton sentry. ktcvs s an oen-source, Layer-7 request dsatcher mlemented as a Lnux kernel module. Request dsatchng for the mddle ter was erformed by an Aache module called mod jk. Fnally, the database ter was based on the Mysql database server. Hostng envronment: We conducted exerments wth the alcatons hosted on two dfferent knds of machnes. The frst hostng envronment conssted of IBM servers (model BU) wth 662 MHz rocessors and 256MB RAM connected by 1Mbs ethernet. The second settng, used for exerments reorted n Secton 7, had Dell servers wth 2.8GHz rocessors and 512MB RAM nterconnected usng ggabt ethernet. Ths served to verfy that our model was flexble enough to cature alcatons runnng on dfferent tyes of machnes. Fnally, the workload generators were run on machnes wth Pentum-III rocessors wth seeds 45MHz- 1GHz and RAM szes n the range MB. The workload generators were always assgned enough machnes so as not to be a bottleneck. All the machnes ran the Lnux kernel. 6.2 Performance Predcton We conduct a set of exerments wth the urose of ascertanng the ablty of our model to redct the resonse tme of mult-ter alcatons. We exerment wth () two knds of alcatons (Rubs and Rubbos), () two dfferent mlementatons of Rubs (based on Java servlets and EJBs), and () dfferent workloads for Rubs. Each of the three alcaton ters are assgned one server. We vary the number of concurrent sessons seen by the alcaton and measure the average resonse tmes of successfully fnshed requests over 3 sec ntervals. Each exerment lasts 3 mn. We comute the average resonse tme and the 95% confdence ntervals from these observatons. Our frst exerment uses Rubs wth a Java servlets-based mddle ter. We use two dfferent workloads W 1: CPU-ntensve on the Java servlets ter, and W 2: CPU-ntensve on the database ter. These were created by modfyng the Rubs clent so that t generated an ncreased fracton of requests that stressed the desred ter. Earler, n Fgure 4(b) we had resented the average resonse tme and 95% confdence ntervals for the workload W 1. Also lotted were the average resonse tmes redcted by our basc model and our model enhanced to handle concurrency lmts. Addtonally, we resent the observed and redcted resdence tmes n Fgure 6(a). Fgure 6(b) shows that the CPU on the Java servlets ter becomes saturated beyond 1 sessons for ths workload. As already exlaned n Secton 4, the basc model fals to cature the resonse tmes for workloads hgher than about 1 sessons due to an ncrease n the fracton of requests that arrve at the Aache and servlets ters only to be droed because of the ters oeratng at ther concurrency lmts. We fnd that our enhanced model s able to cature the effect of droed requests at these hgh workloads and contnues to redct resonse tmes well for the entre workload range. Fgure 7 lots the resonse tmes, the resdence tmes, and the server CPU utlzatons for servlets-based Rubs subjected to the workload W 2 wth varyng number of sessons. As shown n Fgure 7(c), the CPU on the database server s the bottleneck resource for ths workload. We fnd that our basc model catures resonse tmes well. The redcted resonse tmes are wthn the 95% confdence nterval of the observed average resonse tme for the entre workload range. Next, we reeat the exerment descrbed above wth Rubs based on an EJB-based mddle ter. Our results are resented n Fgure 8. Agan, our basc model catures the resonse tme well untl the concurrency lmts at Aache and JBoss are reached. As the number of sessons grows beyond ths ont, ncreasngly large fractons of requests are droed, the request throughut saturates, and the resonse tme of requests that fnsh successfully shows a flat trend. Our enhancement to the model s found to cature ths effect well.

9 Avg. res. tme (msec) Observed Basc Model Enhanced Model Num. smult. sessons (a) Resonse tme Avg. CPU usage (%) Aache JBoss Mysql Num. smult. sessons (b) CPU utlzatons Fgure 8: Rubs based on EJB: bottleneck at CPU of mddle ter. The concurrency lmts for the Aache Web server and the Java servlets contaner were set to be 15 and 75, resectvely. Avg. res. tme (msec) Observed Basc Model Enhanced Model Num. smult. sessons (a) Resonse tme Avg. CPU usage (%) Aache Tomcat Mysql Num. smult. sessons (b) CPU utlzatons Fgure 9: Rubbos based on Java servlets: bottleneck at CPU of mddle ter. The concurrency lmts for the Aache Web server and the Java servlets contaner were set to be 15 and 75, resectvely. Fnally, we reeat the above exerment wth the Rubbos alcaton. We use a Java servlets based mddle ter for Rubbos and subject the alcaton to the workload W 1 that s CPU-ntensve on the servlets ter. Fgure 9 resents the observed and redcted resonse tmes as well as the server CPU utlzatons. We fnd that our enhanced model redcts resonse tmes well over the chosen workload range for Rubbos. 6.3 Query Cachng at the Database Recent versons of the Mysql server feature a query cache. When n use, the query cache stores the text of a SELECT query together wth the corresondng result that was sent to the clent. If the dentcal query s receved later, the server retreves the results from the query cache rather than arsng and executng the query agan. Query cachng at the database has the effect of reducng the average servce tme at the database ter. We conduct an exerment to determne how well our model can cature the mact of query cachng on resonse tme. We subject Rubbos to a workload consstng of 5 smultaneous sessons. To smulate dfferent degrees of query cachng at Mysql, we use a feature of Mysql queres that allows the ssuer of a query to secfy that the database server not use ts cache for servcng ths query 5. We modfed the Rubbos servlets to make them request dfferent fractons of the queres wth ths oton. For each degree of cachng we lot the average resonse tme wth 95% confdence ntervals n Fgure 1. As exected, the observed resonse tme decreases steadly as the degree of query cachng ncreases the average resonse tme s nearly 14 msec wthout query cachng and reduces to about 1 msec when all the queres are cached. In Fgure 1 we also lot the av- 5 Secfcally, relacng a SELECT wth SELECT SQL NO CACHE ensures that Mysql does not cache ths query. Avg. res. tme (msec) Observed Model Degree of query cachng Fgure 1: Cachng at the database ter of Rubbos. erage resonse tme redcted by our model for dfferent degrees of cachng. We fnd that our model s able to cature well the mact of the reduced query rocessng tme wth ncreasng degrees of cachng on average resonse tme. The redcted resonse tmes are found to be wthn the 95% confdence nterval of the observed resonse tmes for the entre range of query cachng. 6.4 Multle Sesson Classes We created two classes of Rubs sessons usng the workloads W 1 and W 2 resectvely. Recall that the requests n these classes have dfferent servce tme requrements at dfferent ters W 1 s CPU-ntensve on the Java servlets ter whle W 2 s CPU ntensve on the database ter. We conduct two sets of exerments, each of whch nvolves keeng the number of sessons of one class fxed at 1 and varyng the number of sessons of the other class. We then comute the er-class average resonse tme redcted by the multclass verson of our model (Secton 5). We lot the observed and redcted resonse tmes for the two classes n Fgure 11. Whle the redcted resonse tmes closely match the observed values for the frst exerment, n the second exerment (Fgure 11(b)), we observe that our model underestmates the resonse tme for class 1 for 5 sessons we attrbute ths to an naccurate estmaton of the servce tme of class 1 requests at the servlets ter at ths load. 7. APPLICATIONS OF THE MODEL In ths secton we demonstrate some alcatons of our model for managng resources n a data center. 7.1 Dynamc Caacty Provsonng Dynamc caacty rovsonng s a useful technque for handlng the mult-tme-scale varatons seen n Internet workloads. The goal of dynamc rovsonng s to dynamcally allocate suffcent caacty to the ters of an alcaton so that ts resonse tme needs can be met even n the resence of the eak workload. Two key comonents of a dynamc rovsonng technque are: () redctng the workload of an alcaton, and () determnng the caacty needed to serve ths redcted workload. The former roblem has been addressed n aers such as [6]. The workload estmates made by such redctors can be used by our model to address the ssue of how much caacty to rovson. Observe that the nuts to our model-based rovsonng technque are the workload characterstcs, number of sessons to be servced smultaneously, and the resonse tme target, and the desred outut s a caacty assgnment for the alcaton. We start wth an ntal assgnment of one server to each ter. We use the MVA algorthm to determne the resultng average resonse tme as descrbed n Sectons 3, 4,

10 Avg. res. tme (msec) Observed for class 1 Observed for class 2 Predcted for class 1 Predcted for class 2 Avg. res. tme (msec) Observed for class 1 Observed for class 2 Predcted for class 1 Predcted for class Num. class 2 sessons (a) Ten class-1 sessons Num. class 1 sessons (b) Ten class-2 sessons Fgure 11: Rubs servng sessons of two classes. Sessons of class 1 were generated usng workload W 1 whle those of class 2 were generated usng workload W 2. and 5. In case ths s worse than the target, we use the MVA algorthm to determne, for each relcable ter, the resonse tme resultng from the addton of one more server to t. We add a server to the ter that results n the most mrovement n resonse tme. We reeat ths tll we have an assgnment for whch the redcted resonse tme s below the target ths assgnment yelds the caacty to be assgned to the alcaton s ters 6. The above rovsonng rocedure has a tme comlexty of O(kMN), where k s the number of servers that the alcaton s eventually assgned, M s the the number of ters, and N s the number of sessons. Snce rovsonng decsons are tycally made over erods of tens of mnutes or hours, ths overhead s ractcally feasble. We conduct an exerment to demonstrate the alcaton of our model to dynamcally rovson Rubs confgured usng Java servlets at ts mddle ter. We assume an dealzed workload redctor that can accurately forecast the workload for the near future. We generated a 1-hour long sesson arrval rocess based on a Web trace from the 1998 Soccer World Cu ste [2]; ths s shown n Fgure 12(a). Sessons are generated accordng to ths arrval rocess usng workload W 1. We mlemented a rovsonng unt that nvokes the modelbased rocedure descrbed above every 1 mn to determne the caacty requred to handle the workload durng the next nterval. Our goal was to mantan an average resonse tme of 1 sec for Rubs requests. Snce our model requres the number of smultaneous sessons as nut, the rovsonng unt converted the eak rate durng the next nterval nto an estmate of the number of smultaneous sessons for whch to allocate caacty usng Lttle s Law [8] as N = Λ d, where Λ s the eak sesson arrval rate durng the next nterval as gven by the redctor and d s the average sesson duraton. The rovsonng unt ran on a searate server. It mlemented scrts that remotely log on to the alcaton sentry and the dsatchers for the affected ters after every re-comutaton to enforce the newly comuted allocatons. The concurrency lmts of the Aache Web server and the Tomcat servlets contaner were both set to 1. We resent the workng of our rovsonng unt and the erformance of Rubs n Fgure 12(b). The rovsonng unt s successful n changng the caacty of the servlets ter to 6 Note that our current dscusson assumes that t s always ossble to meet the resonse tme target by addng enough servers. Sometmes ths may not be ossble (e.g., due to the workload exceedng the entre avalable caacty, or a non-relcable ter becomng saturated) and we may have to emloy admsson control n addton to rovsonng. Ths s dscussed n Secton 7.2. match the workload recall that workload W 1 s CPU ntensve on ths ter. The sesson arrval rate goes u from about 1 sess/mn at t = 2 mn to nearly 3 sess/mn at t = 4 mn. Corresondngly, the request arrval rate ncreases from about 15 req/mn to about 42 req/mn. The rovsonng unt ncreases the number of Tomcat relcas from 2 to a maxmum of 7 durng the exerment. Further, at t = 3 mn, the number of smultaneous sessons durng the ucomng 1 mn nterval s redcted to be hgher than the concurrency lmt of the Aache ter. To revent new sessons beng droed due to the connecton lmt beng reached at Aache, a second Aache server s added to the alcaton. Thus, our model-based rovsonng s able to dentfy otental bottlenecks at dfferent ters (connectons at Aache and CPU at Tomcat) and mantan resonse tme targets by addng caacty arorately. We note that the sngle-ter models descrbed n Secton 2.3 wll only be able to add caacty to one ter and wll fal to cature such changng bottlenecks. 7.2 Sesson Polcng and Class-based Dfferentaton Internet alcatons are known to exerence unexected surges n ther workload, known as flash crowds [21]. Therefore an mortant comonent of any such alcaton s a sentry that olces ncomng sessons to an alcaton s server ool ncomng sessons are subjected to admsson control at the sentry to ensure that the contracted erformance guarantees are met; excess sessons are turned away durng overloads. In an alcaton suortng multle classes of sessons, wth ossbly dfferent resonse tme requrements and revenue schemes for dfferent classes, t s desrable to desgn a sentry that, durng a flash crowd, can determne a subset of sessons admttng whch would otmze a meanngful metrc. An examle of such a metrc could be the overall exected revenue generated by the admtted sessons whle meetng ther resonse tme targets (ths constrant on resonse tmes wll be assumed to hold n the rest of our dscusson wthout beng stated). Formally, gven L sesson classes, C 1,, C L, wth u to N sessons of class C and usng overall revenue as the metrc to be otmzed, the goal of the sentry s to determne an L-tule (N1 admt,, ) such that n N (1 L), rev (N admt ) rev (n ) N admt L where rev (n ) denotes the revenue generated by n admtted sessons of C.

11 arrvals er mn Tme (mn) (a) Arrvals Avg res tme (msec) Res. tme Num. Aache Num. Tomcat Tme (mn) (b) Server allocs. and res. tme Number of servers Fgure 12: Model-based dynamc rovsonng of servers for Rubs. Our mult-class model descrbed n Secton 5 rovdes a flexble rocedure for realzng ths. Frst observe that the nuts to ths rocedure are the workload characterstcs of varous classes and the caacty assgned to the alcaton ters, and the desred outut s the number of sessons of each class to admt. In theory, we could use the mult-class MVA algorthm to determne the revenue yelded by every admssble L-tule. Clearly ths would be comutatonally rohbtve. Instead, we use a heurstc that consders the sesson classes n a non-ncreasng order of ther revenue-ersesson. For the class under consderaton, t adds sessons tll ether all avalable sessons are exhausted, or addng another sesson would cause the resonse tme of at least one class, as redcted by the model, to volate ts target. The outcome of ths rocedure s an L-tule of the number of sessons that can be used by the olcer to make admsson control decsons. We now descrbe our exerments to demonstrate the workng of the sesson olcer for Rubs. We confgured the servlets verson of Rubs wth 2 relcas of the servlets ter. Smlar to Secton 6.4, we chose W 1 and W 2 to construct two sesson classes C 1 and C 2 resectvely. The resonse tme targets for the two classes were chosen to be 1 sec and 2 sec; the revenue yelded by each admtted sesson was assumed to be $.1 and $1 resectvely. We assume sesson duratons of exactly 1 mn for llustratve uroses. We create the followng flash crowd scenaros. We assume that 15 sessons of C 1 and 1 sessons of C 2 arrve at t = ; 5 sessons each of C 1 and C 2 are assumed to arrve at t = 1 mn. Fgure 13(a) resents the workng of our model-based olcer. At t =, based on the rocedure descrbed above, the olcer frst admts all 1 sessons of the class wth hgher revenue-er-sesson, namely C 2; t then roceeds to admt as many sessons of C 1 as t can (9) whle keeng the average resonse tmes under target. At t = 1 mn, the olcer frst admts as many sessons of C 2 as t can (21); t then admts 5 sessons of C 1 admttng more would, accordng to the model, cause the resonse tme of C 2 to be volated. We fnd from Fgure 13(a) that the resonse tme requrements of both the classes are met durng the exerment. We make two addtonal observatons: () durng [, 1] mn, the resonse tme of C 2 s well below ts target of 2 sec ths s because there are only 1 sessons of ths class, less than the caacty of the database ter for the desred resonse tme target; snce the 9 sessons of C 1 stress manly the servlets ter (recall the nature of W 1 and W 2), they have mnmal mact on the resonse tme of C 2 sessons, whch manly exercse the database ter, and () durng (1, 2] mn, the resonse tme of C 1 s well below ts target of 1 sec ths s because the olcer admts only 5 C 1 sessons; the servlets ter s lghtly loaded snce the C 2 sessons do not stress t, and therefore the C 1 sessons exerence low resonse tmes. Fgure 13(b) demonstrates the mact of admttng more sessons on alcaton resonse tme. At t =, the olcer admts excess C 1 sessons t admts 14 and 1 sessons resectvely. We fnd that sessons of C 1 exerence degraded resonse tmes (n excess of 2 sec as oosed to the desred 1 sec). Smlarly, at t = 1 mn, t admts excess C 2 sessons t admts 5 and 31 sessons resectvely. Now sessons of C 2 exerence resonse tme volatons. Observe that admttng excess sessons of one class does not cause a ercetble degradaton n the erformance of the other class because they exercse dfferent ters of the alcaton. 8. CONCLUSIONS In ths aer we resented an analytcal model for mult-ter Internet alcatons. Our model s based on usng a network of queues to reresent how the ters n a mult-ter alcaton cooerate to rocess requests. Our model s () general enough to cature Internet alcatons wth an arbtrary number of heterogeneous ters, () s nherently desgned to handle sesson-based workloads, and () can account for alcaton dosyncrases such as cachng effects, the resence of multle classes of sessons, and lmts on the amount of concurrency at each ter. The model arameters are easy to measure and udate. We valdated the model usng two oen-source mult-ter alcatons runnng on a Lnuxbased server cluster. Our exerments demonstrated that our model fathfully catures the erformance of these alcatons for a varety of workloads and confguratons. We demonstrated the utlty of our model n managng resources for Internet alcatons under varyng workloads and shftng bottlenecks. As art of future work, we lan to nvestgate the sutablty of our model for caturng more dverse workloads (e.g., IO-ntensve at certan ters) and to desgn enhancements to handle these. Another drecton s to extend our model to handle other knds of schedulng dsclnes (such as roortonal-share schedulng) at the alcaton servers. Fnally, our model does not cature multle sesson classes n the resence of concurrency lmts. We lan to enhance our model to cature the smultaneous resence of these two artfacts. 9. REFERENCES [1] T. Abdelzaher, K. G. Shn, and N. Bhatt. Performance Guarantees for Web Server End-Systems: A Control-Theoretcal Aroach. IEEE Transactons on Parallel and Dstrbuted Systems, 13(1), Jan. 22.

12 Avg. res. tme (msec) Res. tme C1 Res. tme C Tme (mn) (a) Model-based olcng Avg. res. tme (msec) Res. tme C1 Res. tme C Tme (mn) (b) Polcer admts more than caacty Fgure 13: Maxmzng revenue va dfferentated sesson olcng n Rubs. The alcaton serves two classes of sessons. [2] M. Arltt and T. Jn. Workload Characterzaton of the 1998 World Cu Web Ste. Techncal Reort HPL R1, HP Labs, [3] A. Chandra, W. Gong, and P. Shenoy. Dynamc Resource Allocaton for Shared Data Centers Usng Onlne Measurements. In Proceedngs of Eleventh Internatonal Worksho on Qualty of Servce (IWQoS 23), June 23. [4] R. Doyle, J. Chase, O. Asad, W. Jn, and A. Vahdat. Model-Based Resource Provsonng n a Web Servce Utlty. In Proceedngs of the 4th USITS, Mar. 23. [5] Dynaserver roject. htt://comsc.rce.edu/cs/ Systems/DynaServer/. [6] J. Hellersten, F. Zhang, and P. Shahabuddn. An Aroach to Predctve Detecton for Servce Management. In Proceedngs of the IEEE Intl. Conf. on Systems and Network Management, [7] A. Kamra, V. Msra, and E. Nahum. Yaksha: A Controller for Managng the Performance of 3-Tered Webstes. In Proceedngs of the 12th IWQoS, 24. [8] L. Klenrock. Queueng Systems, Volume 1: Theory. John Wley and Sons, Inc., [9] Kernel TCP Vrtual Server. htt:// software/ktcvs/ktcvs.html. [1] E. Lazowska, J. Zahorjan, G. Graham, and K. Sevck. Quanttatve System Performance. Prentce Hall, [11] R. Levy, J. Nagarajarao, G. Pacfc, M. Sretzer, A. Tantaw, and A. Youssef. Performance Management for Cluster Based Web Servces. In IFIP/IEEE Eghth Internatonal Symosum on Integrated Network Management, volume 246, ages , 23. [12] D. Menasce. Web Server Software Archtectures. In IEEE Internet Comutng, volume 7, November/December 23. [13] Oracle9. htt:// roducts/oracle9. [14] S. Ranjan, J. Rola, H. Fu, and E. Knghtly. Qos-drven Server Mgraton for Internet Data Centers. In Proceedngs of the Tenth Internatonal Worksho on Qualty of Servce (IWQoS 22), May 22. [15] M. Reser and S. Lavenberg. Mean-Value Analyss of Closed Multchan Queung Networks. In Journal of the Assocaton for Comutng Machnery, volume 27, ages , 198. [16] Sysstat ackage. htt://freshmeat.net/rojects/sysstat. [17] L. Slothouber. A Model of Web Server Performance. In Proceedngs of the 5th Internatonal World Wde Web Conference, [18] B. Urgaonkar and P. Shenoy. Cataclysm: Handlng Extreme Overloads n Internet Servces. In Proceedngs of the 23rd Annual ACM SIGACT-SIGOPS Symosum on Prncles of Dstrbuted Comutng (PODC), July 24. [19] B. Urgaonkar, G. Pacfc, P. Shenoy, M. Sretzer, and A. Tantaw. An Analytcal Model for Mult-ter Internet Servces and ts Alcatons. Techncal reort TR4-99, Deartment of Comuter Scence, Unversty of Massachusetts, October 24. [2] D. Vllela, P. Pradhan, and D. Rubensten. Provsonng Servers n the Alcaton Ter for E-commerce Systems. In Proceedngs of the 12th IWQoS, June 24. [21] M. Welsh and D. Culler. Adatve Overload Control for Busy Internet Servers. In Proceedngs of the 4th USITS, March 23.