Little s Law & Bottleneck Law

Transcription

1 Lttle s Law & Bottleneck Law Dec 20 I professonals have shunned performance modellng consderng t to be too complex and napplcable to real lfe. A lot has to do wth fear of mathematcs as well. hs tutoral provdes the foundaton of performance modellng. Usng elementary mathematcs two fundamental laws are dscussed. Lttle s Law s the most popular law n performance modellng and s applcable to all I systems. he bottleneck law s also very applcable and though used qualtatvely, t s seldom appled quanttatvely. hs tutoral shows the applcablty of both these laws to a varety of stuatons that are relevant to the I ndustry. opyrght () 20 Rajesh Mansharaman Permsson s granted to copy, dstrbute and/or modfy ths document under the terms of the GNU Free Documentaton Lcense, Verson.3 or any later verson publshed by the Free Software Foundaton; wth no Invarant Sectons, no Front-over exts, and no Back-over exts. A copy of the lcense s ncluded n the secton enttled "GNU Free Documentaton Lcense".

2 Lttle s Law & Bottleneck Law Rajesh.Mansharaman@swperfengg.com wo smple laws are ntroduced n ths document, whch have wdespread applcablty. hese fundamental paradgms should suffce for a lot of quanttatve analyss of I systems, as wll be shown through a set of examples provded and a set of exercses at the end of the document. Before we get nto the laws let us ntroduce the basc set of terms and notaton that wll be used to derve the laws. hereafter we get n to Lttle s Law and then the Bottleneck Law. Lttle s Law provdes a relatonshp between response tme, number n the system and throughput. he Bottleneck Law shows where the system wll bottleneck n terms of system throughput and usng t we can derve useful response tme and throughput bounds.. Defntons of Fundamental erms We defne seven fundamental terms n ths secton, three are wdely used n the software ndustry today, and the other four wll be useful for the purpose of dscusson. Response tme, throughput, and number of concurrent users are popular terms used by software and I systems professonals. he other four terms of nterest to us are thnk tme, servce tme, vst count, and demand. By default all notaton that we use wll stand for averages, unless otherwse stated. hus for example, R wll be used to denote average response tme.. System, Resource, Entty, and Work onservng System he fundamental terms to be defned are n the context of a system. As shown n Fgure, a system s composed of a set of sub-systems, the most granular of whch s a sngle resource. Enttes flow through the system requestng servces from varous resources. For example, a data centre for a large bank s a system and a bankng transacton such as a depost s an entty. he bankng transacton vsts varous system resources such as network, PUs of the web, applcaton, and database servers, as well as dsks attached to the database server. In ths context, a network swtch s also a system, where the enttes flowng through t are network packets. It s assumed that a resource can servce only one request at any gven nstant of tme. hus, when a PU s servcng a request at any nstant of tme, even f t s n tme slcng mode, t s not servcng any other request at that pont n tme. Note that ths dscusson s only for one resource at an nstant of tme. A system s sad to be work conservng f no work s generated wthn the system and no work gets destroyed wthn the system. For example, consder a bank, whch provdes for deposts and wthdrawal transactons. hen work conservng means that a depost or a wthdrawal transacton s not generated unless there s a vald customer request, and once a transacton s accepted t s not dened by the bank.

3 Fgure : System and Entty.2 Response me Response tme of an entty s the tme dfference between entry and ext of the entty from the system. hus, response tme of a depost transacton n the data centre s the tme the transacton exts from the data centre network mnus the tme t enters n to the data centre network. However, f we consder end user response tme, then we need to extend our system boundary to the end user termnals, so that the wde area network tme as well as the tme spent at the user termnal s counted. he symbol R wll be used to denote average response tme. If we observe the system for a perod of tme, and enttes leave the system durng that tme, wth response tmes R,...,R then we can compute the average response tme R as.3 hroughput R = (R + R R ) / hroughput of a system s ts rate of processng enttes. hus number of bankng transactons per day, number of web hts per second, number of I/Os per second, as well as network bandwdth n Kbps or Mbps, are all measures of throughput. he symbol X wll be used to denote throughput. If we observe the system for a perod of tme, and enttes leave the system durng that tme, then we have throughput as: X hroughput and response tme are the two most mportant metrcs for I systems. A hgh throughput means more busness transactons and a low response tme means better end-user satsfacton..4 Number n the System, hnk me, and Number of oncurrent Users he number of enttes beng served n any system s a very useful measure for system capacty plannng. We nterchangeably use the terms number n the system and number of enttes n the system. hs s a functon of tme, snce at 2

4 dfferent ponts n tme the number n the system wll vary. If the system s work conservng, then we have number n the system at any tme t as: N(t) = Number of Arrvals up to tme t Number of Departures up to tme t he symbol N wll be used to denote average number n the system. If we observe the system for a perod of tme, then we have average number n the system as : N 0 N( t) dt A very wdely used term n the I ndustry s the number of concurrent users. In ths context, we are speakng of end-users. he user s actvtes wll n turn mpact the rate at whch enttes come n to the system. We wll use the term thnk tme to denote the tme between recevng a response to a request and submttng the next request. he end user could be revewng the response, or could be enterng data n to the next request s screens, or could have stepped out and come back. Essentally, the user s stll part of the system but has not submtted any entty for processng. In some lterature thnk tme s also referred to as sleep tme. We wll use the symbol Z to denote average thnk tme. As shown n Fgure 2, the number of concurrent users n the system s essentally the number of end-users who are usng the system; they could, for example, be ether thnkng or could have enttes submtted for processng. Fgure 2: oncurrent Users.5 Servce me, Vst ount, and Demand Servce tme s the tme spent by a resource n servcng a sngle request by an entty. For example, a sngle bankng transacton makes 5 requests to a web server wth an average of 5ms PU tme per request, 2 requests to an applcaton server wth an average of 0ms PU tme per request, 4 requests to a database server o fnd out average of a functon, one needs to take a number of samples, add them up and dvde the sum by the number of samples. For a functon whch changes rapdly wth tme, the number of samples needs to enough to cover all changes. hus when the number of samples tends to nfnty, the summaton becomes an ntegral. 3

5 wth an average of 20ms PU tme per request, and 0 requests to the dsk subsystem wth an average of 5ms dsk servce tme per request. Note that the servce tmes n ths example are tme spent n servcng the request, and they do not ncludng queung tme or wat tme at the resource, whch forms part of response tme. In other words, servce tme at a resource can be thought of as response tme at the resource under dle condtons. We use the symbol S to denote average servce tme. In the prevous example we saw that a sngle transacton makes multple vsts to sub-systems and resources. he average number of vsts to a resource s called the vst count of that entty at the resource. Note that vst count by defnton s an average. Also note that vst count s a relatve number. In the example above, one bankng transacton makes 4 requests to the database server and 0 requests to the dsk subsystem. hus the vst count s 4 at the database server and 0 at the dsk subsystem. hs s relatve to the bankng transacton. At the dsk subsystem the vst count relatve to the database s 2.5 (0/4). Vst count can also be a fracton, whch less than one. In the example above, f we have 8 PUs at the database server, then the vst count per PU s 4/8 = 0.5. We use the symbol V to denote vst count. Whether we make 4 requests to the database server wth servce tme 20ms per request, or request wth servce tme 80ms, the total servce demand at the database server remans the same, that s, 80ms. hus the average demand at a resource s the product of average servce tme at that resource and the vst count at that resource. he symbol D s used to denote average demand. hus at each resource n the system the average demand s: D V S.6 Summary of Notaton able summarzes the notaton we wll be usng n the rest of ths document. able : Notaton Symbol D N R S V X Z Descrpton Average Demand Average Number n System Average Response me Average Servce me Vst ount hroughput Average hnk me 4

6 2. Lttle s Law Lttle s Law, named after John D.. Lttle, states that the average number n a work conservng system equals the product of the system throughput and the average response tme. In ths secton, the proof for Lttle s Law wll be worked out, and extended to end-user systems. he smplcty and hgh applcablty of ths law has resulted n t beng wdely used n performance modellng. 2. Lttle s Law for any system he technque used n the proof s operatonal analyss. Let us observe arrvals and departures n a system. As shown n Fgure 3, let A(t) be the total number of arrvals up to tme t, and D(t) be the total number of departures up to tme t. Note that A(t) and D(t) are both non-decreasng functons. We also assume that each of them can be ncreased only n steps of unt. In other words, the probablty of two or more arrvals at exactly the same tme and of two or more departures at exactly the same tme s zero. hs s not a lmtaton snce we model two smultaneous arrvals as comng n at tme t and tme t+t, where t s an nfntesmally small nterval of tme. Snce the system s work conservng we cannot have more departures than arrvals, and hence D(t) A(t). By defnton of number n the system at any pont n tme t, N(t) (see Secton.4) we have N(t) = A(t) D(t) Fgure 3: Arrvals and Departures n a Work onservng System We are nterested n determnng the average number n the system, whch from Secton.4 s gven by: N 0 N( t) dt he ntegral of N(t) s the area under the functon, or the area between the functons A(t) and D(t), as shown n Fgure 4. 5

7 Fgure 4: Area Between Arrvals and Departures he shaded area n Fgure 4 s the sum of areas of the rectangles shown. Each rectangle s of heght (snce we assumed that no two arrvals or no two departures happen at exactly the same nstant of tme). he wdth of any rectangle s the dfference between the tme of the th departure, d and the tme of the th arrval, a. If the system has the same sequence of departures as the sequence of arrvals, or n other words t s a frst-come-frst-serve (FFS) system then d a s the response of tme job, denoted by R. We wll proceed wth the FFS assumpton and then prove the law can be extended for any other sequence of departures. We now have the area of rectangle as x R = R. herefore the average number n the system s: N 0 N( t) dt = R Here s the number of completons up to tme. In realty, equaton () wll hold for the lmt snce we are not consderng partal response tme of any request beng served at tme. herefore, we need to assume that the nterval of tme s large enough compared to ndvdual response tmes. We can rewrte equaton () by dvdng and multplyng wth to get: () R R (2) From Sectons.3 and.2, we can see that the frst term on the rght hand sde s the system throughput and the second term s the average response tme. We therefore have: R ombnng equatons () and (3) we get: X R (3) 6

8 N = X R Lttle s Law (4) o prove Lttle s Law we had assumed FFS sequencng n the system. Let s now show that Lttle s Law apples for any type of sequencng. he key to Lttle s Law s equaton () n whch we have assumed that the shaded area n Fgure 4 s the sum of the response tmes. Whle ths s true for FFS, why does t hold for any sequence of departures? Even f the sequence of departures s not FFS, there wll be a one-to-one parng between arrvals and departures. Fgure 5 shows a sample mappng of arrvals to departures. Fgure 5: Sample mappng of arrvals to departures Now let us renumber the departures as per ther arrval sequence. If we order the enttes n ther sequence of arrvals, then let d denote the departure tme of the th entty. hus d s the departure tme of the frst entty, whch by Fgure 5, s nothng but d 2, as per the one s to one mappng of arrvals and departures. Fgure 6, shows the departure tmes of Fgure 5 renumbered as per the arrval sequence of enttes. Fgure 6: Renumberng of departures By ths renumberng we therefore have response tme of entty as (d a ): R = (d a ) (5) omng back to the area of the shaded regon n Fgure 4, we see that the area s the sum of areas of rectangles each of heght and wdth (d a ): 0 N( t) dt ( d a ) (6) 7

9 he summaton on the rght hand sde can be expressed as: ( d a ) d a d ' a ( d ' a ) (7) ombnng equatons (6), (7), and (5) we get 0 ' N( t) dt ( d a ) ( d a ) R (8) On account of equaton (8), we have equaton () holdng for any sequence of departures, and thus Lttle s Law holds for any sequence of departures. o summarze, we have just proved that average number n a system s the product of the system throughput and the average response tme. N = X R 2.2 Lttle s Law for end-user systems onsder ths: f we have an I system wth,000 concurrent users, and average response tme of 2 seconds, then can we say that by Lttle s Law the system throughput s 000/2 = 500 requests/sec? hs wll not be correct snce there may not be,000 requests n the system. o extend Lttle s Law to end-user systems, consder Fgure 7, whch shows a system whose response tme s of nterest, as well as the end-users of the system who have an average thnk tme Z. Fgure 7: End-User System In Fgure 7, we consder an extended system shown by the dashed lne, whch ncludes the end-users. We have a total of N users n the extended system, each ether thnkng or watng for a response. hus, the total number n ths system s N. he response tme of ths extended system s the sum of the thnk tme and the response tme of the system. Lttle s Law must hold for the extended system as well. We therefore have for end-user systems: N = X (R + Z) (0) 8

10 For the example we started ths subsecton wth, f we have average thnk tme as 8 seconds, then we wll get the throughput as 000/(2+8) = 50 requests/sec. Another smple way to prove Lttle s Law for end-user systems or closed systems s as follows. he cycle tme for a sngle user s (R+Z). herefore the throughput per user s /(R+Z) and therefore the throughput for N users s N/(R+Z). In performance modellng when we look at a sngle resource or subsystem n solaton, we can vew t as an open system and use N = X R, to get the average queue sze or number n that subsystem or resource. hs s also true for very large systems lke google.com where t s mpossble to estmate the number of users, but t s possble to estmate the rate of access, whch means we can treat t as an open system. However, whenever we deal wth a fxed number of users then the closed system model works best. 3. Bottleneck Law and hroughput and Response me Bounds onsder an end-user system as llustrated n Fgure 8. An entty requestng servces of the system vsts several resources, wth a certan vst count and average servce tme. he crcles n the system denote resources, and the tuples shown next to the crcles specfy the vst count and average servce tme at the resources. Fgure 8: Vst ounts and Servce mes As defned n Secton.5, the average demand at a resource s the product of the vst count and the servce tme. For the purpose of our analyss we can equate Fgure 8 to Fgure 9, whch shows the system as a ppelne of resources each havng servce tme equal to demand. In other words, nstead of specfyng that a resource s vsted V tmes wth an average servce tme of S, we specfy that the resource s vsted once wth average demand of D. For the purpose of the bounds derved n ths secton, ths translaton works approprately. 9

11 Fgure 9: Ppelne of demands If we consder any ppelned system such as the one n Fgure 9, the maxmum throughput of the system cannot exceed the throughput at the slowest stage of the ppelne. In the example n Fgure 9, the maxmum throughput of the system s /5. Let the maxmum average demand n the system, across all resources, be denoted by D max : D max{ D } We therefore have the upper bound for system throughput as: max max X () D We refer to ths upper bound on throughput as the Bottleneck Law. In smpler terms you can go faster than the slowest stage n your system. he upper bound holds, regardless of the system workload. When the system saturates ths, the upper bound becomes an equalty. By defnton D max depends on vst counts and servce tmes. D max can be reduced by optmzng the software desgn and mplementaton to reduce servce tmes, or by usng faster PUs or dsks to reduce servce tmes, or by ncreasng the number of resources at a servce centre to reduce the vst count per resource, or by changng the archtecture of the system to reduce vst counts. For example, f database vst counts are hgh, one can ether ncrease the number of PUs or dsks, or ntroduce cachng at the applcaton server n order to reduce the vst counts. From Lttle s Law n equaton (0) we get: N R Z (2) X 0

12 Applyng the upper bound () to equaton (2) we get a lower bound on average response tme: R NDmax Z (3) Bounds () and (3) become equaltes upon system saturaton (unless the system s not work conservng and thrashes after a certan load). We wll see applcatons of these bounds n the next secton. he reader may refer to Lazowska et al. (see bblography at the end of ths chapter) for detaled analyss of bounds ncludng upper bounds on response tme and lower bounds on throughput, as well as bounds on balanced systems. 4. Examples on Lttle s Law and Bottleneck Law A number of real lfe examples on the usage of Lttle s Law and the bounds we have derved n ths chapter are presented n other sectons of ths ste. o sharpen the understandng of the law and the termnology ntroduced n ths secton we present several examples below and also encourage the reader to solve the exercses at the end of ths document. Example : A call centre for a credt card company wshes to plan on the number of employees t should have. he call centre receves 20,000 calls per day whch s expected to grow to 30,000 per day over next 6 months. 75% of the calls occur durng a peak 3 hour perod. he average duraton of a call s 5 mnutes. he call centre would lke ts employees to be 70% utlzed n attendng to calls. What s the number of employees that they should plan for, over the next 6 months? he peak throughput durng next 6 months wll be: X = 75% x 30,000 / 3 hours = 7,500 calls per hour = 25 calls per mnute If the average duraton of a call s 5 mnutes, then the cycle tme at 70% utlzaton per employee s: = 5 mnutes / 0.7 = 7. mnutes herefore average number of employees requred s gven by Lttle s Law: N = X = 25 x 7. = 888 employees Example 2: XYZ.com has wtnessed a surge n ts web ste traffc. It currently servces a maxmum throughput 5 mllon http hts per day across a farm of 0 web servers, each havng 2 PUs. hey have ordered another 5 web servers of 2 PUs each. What wll be the http vst count per PU? What s the throughput that can be servced by the 5 servers, assumng full scalablty at the web ter? Wth 5 more servers comng n, the total number of servers becomes 5 and the total number of PUs becomes 30. An http request can be servced by any of the 30 PUs and hence the http vst count per PU s / 30. Earler the http vst

13 count per PU was / 20. herefore the demand at the web server has come down n proporton to the vst count, that s, by 2/3 rd. herefore the throughput ncreases from 5 mllon hts per day to 5 mllon x 3 / 2 (nversely proportonal to demand) = 7.5 mllon hts per day. Example 3: XYZ.com has now started usng ts ste for revenue generaton. Wth the upgraded set of 5 web servers t servces mllon busness transactons per day. Each busness transacton makes an average of 7.5 http calls and 0 database calls. Assume that 25% of the busness transactons occur n peak one hour wndow and are unformly spread wthn the peak hour. What s the maxmum demand possble at the database server? If the database PU average servce tme per database call s 5 mllseconds, then what s the mnmum number of PUs requred at the database? Peak throughput s: X = 0.25 x mllon / hour = 0.25 mllon busness transactons per hour = 70 busness transactons per second o servce 70 busness transactons per second, the maxmum demand has to be: D max / 70 = 4.3 mllseconds herefore demand at the database ter has to be less than 4.3 mllseconds snce D max s the maxmum demand across all resources. Usng the defnton of demand we get: D database = V database S database 4.3 mllseconds Gven that average servce tme per database call S database s 5 mllseconds we get V database x V database 4.3 / 5 = 2.9 Now let s consder the vst count at the database ter per PU. We have 0 database calls per busness transacton. herefore vst count per database PU s: V database = 0 / (number of database PUs) 2.9 herefore we get: Number of database PUs 0/2.9 herefore there must be a mnmum of 4 PUs at the database. Example 4: In the prevous example, the average servce tme at the database was 5 mllseconds. Assume that for the above throughput ( mllon busness transactons per day) the database PU utlzaton durng peak hours s 90%. By how much should the average servce tme be reduced to brng the database PU utlzaton down to 70%? As can be seen from the exercses secton (Exercse 3), usng Lttle s Law one can derve the followng: 2

14 U = X S Where U s the utlzaton of the resource, X s the throughput, and S the average servce tme at the resource. We are targetng the same throughput of mllon busness transactons per day but now wsh to reduce the utlzaton. Usng ths relatonshp t s clear that average servce tme s drectly proportonal to utlzaton. hus the new average servce tme per database call should be: S = 0.7 / 0.9 x old servce tme = 0.7 / 0.9 x 5 = 3.9 mllseconds Useful Readng Lttle s Law s named after ts nventor and frst appeared n Lttle, J. D.. A Proof of the Queung Formula L = λ W. Operatons Research, 9, (96). It has been wdely used n a number of research papers n manufacturng, computer archtecture, and networks. he bounds for average response tme and throughput, as well as further readng on queueng networks s provded n Lazowska et al. Quanttatve Systems Performance. Prentce Hall, 984. hs book s wdely referenced n the lterature. Snce t s out of prnt t has now been made avalable electroncally from Exercses. A request for proposal (RFP) states that the soluton to be delvered must servce 72,000 transactons per hour, wth 2,000 concurrent users. What s the average cycle tme per user? 2. A web server access log fle (n extended format) shows a throughput of 5,000 hts per mnute wth an average response tme of 200 mllseconds. What s the average number of actve concurrent sessons at the web server? 3. onsder a subsystem wth a number of dentcal resources, for example a PU subsystem or a dsk subsystem. Any request wll undergo a servce tme and a wat tme (f the resources are busy). Now focus only on the resource tself and not the queue for the resource. As shown n the fgure below, draw the system boundary around the resources and apply Lttle s Law. Now prove that utlzaton of the resource U = X S, where X s the (sub)system throughput and S s the average servce tme at the resource. Note that for multple resources at the subsystem U can be more than one. For example, f an 8 PU server s 70% utlzed then U s 5.6 (that s, 0.7 x 8). 3

15 Average Servce me S hroughput X 4. You need to plan on network bandwdth for a large bank that has hundreds of branches all over the country. All branches connect to a central data centre over a wde area network (WAN). Branches come n flavours of small branches (5 users), medum branches (0 users), and large branches (20 users). Assume all users to be statstcally dentcal. he bank s peak throughput at the data centre s 00 transactons per second and there are a total of 5,000 users across all branches. he bank wants an average of second network delay between the branch and the data centre. a. What s the throughput n transactons per second at small, medum, and large branches? b. What s the (applcaton level) bandwdth requred for small, medum, and large branches f the network payload s 4KB per transacton? c. If the average response tme target n the data centre s second, what s the thnk tme per user? Answers to exercses. 00 seconds By Lttle s Law average number n resource subsystem s throughput tmes average response tme n subsystem. Average response tme wthn the resource (outsde of queueng) s average servce tme by defnton. Average number n resource subsystem dvded by number of resources s the percentage utlzaton of the subsystem. Hence U = X S. 4. a) Small branch 0. tps, medum branch 0.2 tps, large branch 0.4 tps b) 3.2Kbps, 6.4Kbps, and 2.8Kbps c) 48 seconds 4