Environmenal Sysems Research Insiue, Inc., 380 New York S., Redlands, CA 92373-8100 USA TEL 909-793-2853 FAX 909-307-3014 Capaciy Planning and Performance Benchmark Reference Guide v. 1.8 Prepared by: ESRI Professional Services Enerprise Implemenaion Services Team Redlands, California July 12, 2009
1 OBJECTIVE...3 2 BENCHMARK USAGE...3 3 BENCHMARK INFORMATION TYPE...3 3.1 BENCHMARK RESULTS...3 3.2 CAPACITY CALCULATOR...4 3.3 BENCHMARK AND MODELED DATA...5 3.4 RELATED INFORMATION...6 4 ESTIMATING CAPACITY USING BENCHMARK DATA...7 4.1 FUNDAMENTAL LAWS, FORMULAS AND DEFINITIONS...7 4.1.1 Sysem Capaciy... 9 4.1.2 CPU Service ime... 10 4.1.3 Queue ime modeling... 11 4.2 ADJUSTING FOR THROUGHPUT AND CONCURRENT USERS RELATIVE DIFFERENCE...11 4.3 ADJUSTING FOR WORKFLOW RELATIVE DIFFERENCE...12 4.4 ADJUSTING FOR TRANSACTION "SIZE" RELATIVE DIFFERENCE...13 4.5 ADJUSTING FOR CPU SPEED RELATIVE DIFFERENCE...14 4.6 EXAMPLE 1: ESTIMATING REQUIRED NUMBER OF CPU CORES...15 4.7 EXAMPLE 2: USING FASTER CPU...16 4.8 EXAMPLE 3: USING FASTER CPU TO MAXIMIZE THROUGHPUT...18 4.9 EXAMPLE 4: USING MORE CPU CORES...19 4.10 EXAMPLE 5: ESTIMATING THE VALUE OF TUNING...21 4.11 EXAMPLE 6: CHANGING FREQUENCY OF USER OPERATIONS...21 4.12 EXAMPLE 7: NETWORK SIZING - CHANGING THE SIZE OF THE MAP DISPLAY...24 4.13 EXAMPLE 8: NETWORK SIZING INCREASING THROUGHPUT...25 5 WALK-THROUGH OF BENCHMARK RESULT SECTIONS...26 5.1 APPLICATION ARCHITECTURE...26 5.2 HARDWARE AND SOFTWARE CONFIGURATION...26 5.3 BENCHMARK RESULTS...26 5.3.1 Performance and Scalabiliy... 27 5.3.2 Key Resource Uilizaion... 28 5.4 CAPACITY PLANNING MODEL INPUT...29 5.4.1 CPU Service Time... 29 5.4.2 Nework Bis/Transacion... 29 6 HELPFUL RESOURCES...30 2
1 Objecive The objecive of his documen is o provide guidance for undersanding and applying he performance benchmarks published in he Implemenaion Gallery sie. This documen describes each secion of a ypical published benchmark and provides addiional informaion regarding mehodology and definiions. 2 Performance Benchmark Usage The ESRI enerprise esing eam conducs and publishes performance benchmarks of represenaive es resuls in he Implemenaion Gallery. Undersanding hese performance benchmarks can help wih, Sysem sizing Capaciy planning model calibraion and inpu Selecion of opimal echnology archiecures Selecion of opimal applicaion archiecure Wihou relevan performance benchmarks, i may be challenging or impossible o conduc effecive capaciy planning. In many cases, esimaing capaciy wih no es daa or using oudaed capaciy models can lead o increased poenial error when planning a soluion. Performance benchmarks provide only a shor descripion and summary of key resuls. To apply hese repors for capaciy planning of differen sysems, refer o he informaion conained in his documen or reques suppor from ESRI Professional Services. 3 Benchmark Informaion Type Found in he Implemenaion Gallery, a performance benchmark repor ypically conains key es resul informaion and focuses on a single es. In many cases, ESRI conducs several variaions of a similar es by changing only one configuraion parameer a a ime. For example, benchmarking an ArcGIS Server map service may involve varying daa source ype (e.g., file geodaabase, shapefile, or ArcSDE) or image compression ype (e.g., JPEG, PNG 8, PNG 24). Since he esed applicaion and mehodology remain he same, variaion resuls are grouped as supplemenal informaion wihin he performance benchmark repor. 3.1 Benchmark Resuls See Walk-hrough of Benchmark Resuls secion ha describes he informaion presened wihin a ypical benchmark repor in more deail. 3
The key par of a performance benchmark repor is Capaciy Planning secion, which provides imporan informaion for sizing esimaes, including: CPU service ime Nework Mbis/ransacion Machine SpecRae (for exrapolaion of resuls for differen hardware) Maximum hroughpu Response ime for a given user load This informaion should be used as an inpu for capaciy planning ools. The benchmark package includes a simple capaciy calculaor, described in he nex secion. 3.2 Capaciy Calculaor Along wih he performance benchmark repor, a simple capaciy calculaor is provided o help calculae he required number of CPUs and nework bandwidh under specified user load, service ime, SpecRae ec. The calculaor is an Excel spreadshee as shown in he following figure. Parameer RT Users Think ST ArcSOC Zoom b Mbis/r b Value Uni 2.80 sec 51.00 users 6.00 sec 0.60 sec 1.81 Mbis/r SpecRae b PerCPU 13.425 SpecRae PerCPU 13.425 %CPU 95 % TH #CPU Mbps 20,864 r/hr 3.63 CPU cores 10.51 Mbps inpu values calculaed values The calculaions are based on he capaciy planning formulas as described in Fundamenal Laws, Formulas and Definiion secion. This ool can be used for capaciy planning he following: Esimaing number of CPU cores required o suppor a defined user load Esimaing required bandwidh o suppor a defined user load Esimae hroughpu based on a defined user load 4
Validaing es resuls hrough comparison wih Capaciy Calculaor values 3.3 Benchmark and Modeled Daa As discussed in he previous secion, performance benchmark resuls can be used o esimae maximum capaciy of a sysem using he Capaciy Calculaor. However, o beer analyze he performance and scalabiliy profile of he benchmark, raw benchmark, modeled daa and comparison chars are provided. This informaion allows an analysis of he enire user load range, no jus a cerain poin of sysem capaciy, as is he case using he Capaciy Calculaor. Beyond his, raw benchmark daa can be used for modeling purposes by alering specific parameers. This allows an invesigaion of he impac of changes such as, 1. Wha happens if a faser CPU is used? 2. Wha happens if more CPU cores are added? 3. Wha happens if ransacion rae varies (hink ime)? The figure below shows he benchmark and modeled daa workshee. Figure 1: Benchmark and Modeled Daa Workshee 5
3.4 Relaed Informaion In addiion o hose iems discussed abovem, he following relaed informaion may be included, if feasible, in a performance benchmark package: Daa and relaed ArcMap documen Tes scrip and load es projec (ypically Visual Sudio Tes Team ediion projec) 6
4 Esimaing Capaciy Using Benchmark Daa Wihou relevan performance benchmarks, i migh be challenging or impossible o conduc an effecive capaciy analysis. In many cases, "ball-parking" or "raionalizing" leads o increased poenial error ha can jeopardize he success of he projec. This secion describes how o effecively uilize he performance benchmarks published in he Implemenaion Gallery sie and how o model a differen sysem. 4.1 Fundamenal Laws, Formulas and Definiions 7
Alhough i is possible o effecively use he Capaciy Calculaor wihou a horough undersanding of he capaciy planning formulas, being familiar wih hese principles will help esimaing he poenial error, validaing he analysis or cusomizing he ool o specific needs. Capaciy planning expressions are as follows: Equaion 1: CPU Service Time # CPU 3600 % CPU ST = TH 100 Equaion 2: Relaive CPU Service Time SpecRaePerCPU b ST = STb SpecRaePerCPU Equaion 3: Response Time as a funcion of CPU Service Time and Queue Time Equaion 4: CPU Sizing # ST TH 100 b CPU = 3600 % CPU SpecRaePerCPU SpecRaePerCPU Equaion 5: Throughpu as a funcion of concurren users and operaion frequency (Response Time + Think Time) TH = RT ST + Q Equaion 6: Throughpu as a funcion of CPU service ime and number of cores Equaion 7: Nework Sizing Mbps Users 3600 = RT + Think = 3600 % CPU # CPU TH = STb 100 TH Mbispr 3600 Where, ST - Service ime RT - Response ime Q - Queue ime b SpecRaePerCPU SpecRaePerCPU b b 8
#CPU - Number of CPU cores %CPU - Percenage of CPU uilizaion (Typically a arge hreshold is se beween 85% and 95%.) TH - Maximum hroughpu Think - Assumed hink ime (sec.) beween ransacions Users - User load b (subscriped) Benchmarked inpus (subscriped) Targe oupus SpecRae SpecRae CINT 2006 benchmarked sysem (See hp://www.spec.org/cpu2006/resuls/cin2006.hml.) 4.1.1 Undersanding Sysem Capaciy Typically, a graphical relaionship may be esablished beween user load or CPU uilizaion and response ime and hroughpu. To undersand his relaionship and deermine capaciy of a sysem, he following graph may be composed: horizonal axis: user load or CPU Uilizaion verical axis: Response Time and Throughpu 9
As we can use, when he sysem has higher uilizaion (e.g. a 40-50%), he response ime sar increasing due o increasing queue ime. I should be noed once he maximum hroughpu is reached, adding more load migh acually degrade hroughpu. The capaciy of he sysem may be defined as one of he following: Maximum hroughpu. Desired response ime, e.g. as per Level of Service agreemen Desired sysem uilizaion. A firs error. 4.1.2 Undersanding CPU Service ime CPU service ime is a measure of he ime required for he CPU o process a reques or ransacion, and is an imporan inpu for capaciy planning. As a general rule, his service ime should remain fairly consan, even as increased load is applied. Performance benchmarks ypically include he CPU service ime for each ier. In order o calculae service ime for a paricular ier, is CPU uilizaion mus be known. Someimes a benchmark is conduced wih each ier on a separae physical machine, allowing uilizaion o be easily found. However, many imes a benchmark is conduced wih all iers on one machine (known as a workgroup configuraion). In his case, each ier s uilizaion may be calculaed by summing he CPU uilizaion of each associaed process. The following are ypical processes encounered: 10
Web: w3wp.exe, lsass.exe ArcGIS Server: arcsom.exe, arcsoc.exe ArcGIS Image Server: ESRIImageServiceProvider.exe Daabase: oracle.exe, sqlserver.exe Deailed process informaion is lised in each performance benchmark s appendix. 4.1.3 Undersanding Queue Time Queue ime is a measure of he ime a reques or ransacion spen waiing o be processed, and is an imporan inpu for capaciy planning. As a general rule, queue ime will increase as load and CPU uilizaion increase, alhough i is someimes considered negligible a lower uilizaion levels ( less han 50%). There are several queuing heory models available, such as M/M/n. In many cases hese generic models provide adequae values for capaciy planning, especially when he sysem is primarily CPU bound. The advanage of performance benchmark daa is ha a cusom model may be developed for each benchmark using regression analysis (a rend line). Typically, an exponenial rend/regression provides he bes fi, as deermined by he coefficien of deerminaion (R 2 ). 4.1.4 Helpful Resources Fundamenal Laws hp://www.cs.washingon.edu/homes/lazowska/qsp/images/chap_03.pdf Lile's Law hp://en.wikipedia.org/wiki/lile_law#use_in_performance_esing_of_compuer_sysems 4.2 Throughpu and Concurren Users Variance Undersanding hroughpu, average concurren users, and peak concurren users on an exising environmen is imporan when beginning o plan for capaciy. In many cases average concurren users may be a very small fracion of he peak concurren users; however i could jus as easily be much higher. In many cases i is difficul o esimae hese values, bu a good place o sar is by, Monioring he exising sysem usage Analyzing proposed business processes and exrapolae average concurren users from peak number of users. For example, via web server logs an analysis could be conduced each hour o undersand usage rends. However, if only a oal monhly usage repor is available, an average usage could be calculaed by dividing by of he monh s operaional hours, (e.g. 8 hours a day, 5 days a week). To calculae hroughpu knowing he concurren number of users and frequency of requess or ransacions, use he hroughpu equaion from he previous secion. For example, if users perform an operaion approximaely every 10 seconds (e.g. 1 sec operaion and 9 seconds hink ime), his would equae o hroughpu 1*3600/10= 360 operaions/hour. 11
Some mission criical applicaions may have high performance and availabiliy requiremens and should be designed for peak concurren users, however his may increase hardware and license coss. For soluions ha have shor peak usage duraions or if emporary performance degradaion is accepable, hese coss may be reduced by designing for average concurren usage. If he hroughpu canno be exrapolaed from he exising usage, i can be esimaed using he Throughpu equaion from he previous secion. The key inpu values are concurren users and frequency of operaions (hink ime) as shown (highlighed in green) in he following figures for benchmark modeled daa and he capaciy calculaor. Think Time Spec_Per_CPU Mbis_per_Tr SOCCPUCoun Benchmark (B) 6 13.425 1.81 4 Targe (T) or Modeled 6 30 1.81 4 Parameer RT Users Think Value Uni 4.00 Sec 1.00 users 6.00 Sec ST ArcSOC Zoom b 0.60 Sec Mbis/r b 1.81 Mbis/r SpecRae b PerCPU 13.425 SpecRae PerCPU 13.425 %CPU 95 % TH 360 r/hr CPU #CPU 0.06 cores Mbps 0.18 Mbps 4.3 Workflow Variance Prior o applying he published ESRI benchmark resuls, i is imporan o undersand differences beween he curren workflow and he performance benchmark so a proper model can be derived. Deermining all deails of he workflow during he early planning phase may be difficul. However, for capaciy planning purposes he focus should primarily be on: Transacions performed frequenly wih a relaively long response ime Bach operaions ransacions wih a long response ime 12
For example, a workflow may be composed of: Transacion 1: o Zoom agains dynamic service. o Think 6 seconds. Transacion 2: o Zoom agains dynamic service. o Think 6 seconds. Transacion 3 o Query agains small daase. o Think 6 seconds. Transacion 4 o Zoom 3 agains dynamic service. o Think 6 seconds. Transacion 5 o Prin agains he dynamic service. For capaciy planning purposes, ransacions performed infrequenly and have comparaively shor response imes may be excluded. In he above example, he query ransacion agains a small daase migh be jus a fracion of a zoom operaion agains he dynamic service. Therefore, he focus should be on he zoom and prin ransacions only. Once a workflow is defined for capaciy planning, review he published benchmarks and selec he relevan Performance Benchmark repor (similar applicaion and workflow) from he Implemenaion Gallery. Even excluding insignifican ransacions, i is possible he workflow will be more complex han one paricular performance benchmark, and several may be required. Gahering he CPU service imes for he various ransacions from he respecive benchmarks, he complex workflow may be pieced ogeher. This is discussed in he CPU Service Time secion. 4.4 Transacion "Size" Variance As menioned in he Workflow Variance secion, ransacions define he core unis of work in a workflow. Each ransacion will have a differen size, which can impac how much nework bandwidh is required o suiably ranspor Transacion "size" ypically is relaed o he following; each can have a significan impac: Daa sources ArcMap documens Number of feaures and layers reurned (e.g., zoom, search) Image compression Map sizes Caching To exrapolae from published benchmark resuls, review and accoun for poenial performance facor differences as described in Performance and Scalabiliy secion. 13
Once he final size has been undersood, i may be adjused wihin he modeled capaciy resul or as shown below in he sample Capaciy Calculaor. RT Users Think 4.00 sec 1.00 users 6.00 sec ST ArcSOC Zoom b 0.60 sec Mbis/r b 1.81 Mbis/r SpecRae b PerCPU 13.425 SpecRae PerCPU 13.425 %CPU 95 % TH 360 r/hr CPU #CPU 0.06 cores Mbps 0.18 Mbps 4.5 Adjusing for CPU Speed Relaive Difference This difference can be measured as a raio of SpecRae CINT 2006 base per core as per hp://www.spec.org/cpu2006/resuls/cin2006.hml published benchmarks. I should be noed ha under a ligh uder load,he CPU speed, no he number of cores, is he primary conribuor o sysem performance (measured as a response ime). For example, a ypical ESRI benchmark CPU is PowerEdge 1950 (Inel Xeon processor 5160, 3.00 GHz), SpecRae/Core = 53.7/4 = 13.425 PowerEdge 1950 III (Inel Xeon E5450, 3.00 GHz), SpecRae/Core = 109/8 = 13.625 The following able is an excerp from a published SpecRae CINT 2006 benchmarked sysem. See hp://www.spec.org/cpu2006/resuls/cin2006.hml. Hardware Vendor Sysem Resul Baseline # Cores Dell Inc. PowerEdge 1950 (Inel Xeon 5140, 2.33 GHz) 0 45.9 4 Dell Inc. PowerEdge 1950 (Inel Xeon 5150, 2.66 GHz) 0 49.8 4 Dell Inc. PowerEdge 1950 (Inel Xeon 5160, 3.00 GHz) 0 53.7 4 Dell Inc. PowerEdge 1950 (Inel Xeon E5335, 2.00 GHz) 0 69 8 Dell Inc. PowerEdge 1950 III (Inel Xeon E5410, 2.33 GHz) 112 92.3 8 Dell Inc. PowerEdge 1950 III (Inel Xeon E5420, 2.50 GHz) 112 98 8 Dell Inc. PowerEdge 1950 III (Inel Xeon E5430, 2.66 GHz) 124 100 8 Dell Inc. PowerEdge 1950 III (Inel Xeon E5450, 3.00 GHz) 125 109 8 14
If all oher facors are consan, for a CPU-bound soluion, he CPU service ime measured from one sysem can be exrapolaed using Relaive CPU Service Time equaion in he previous secion. In our case, he raio is 13.425/13.625 = 0.985. I can be concluded ha even hough differen server ypes are used, he relaive CPU difference is marginal. Users are ofen ineresed in relaive performance (response ime difference) beween wo sysems. In case of low CPU uilizaion(less han 50%) queue ime can be considered negligible. However, for a CPU- bound sysem response ime (RT) can be esimaed from a slower o a faser sysem using he following formula: SpecRaePerCPU b RT RTb SpecRaePerCPU 4.6 Example 1: Esimaing required number of CPU cores A hosing company needs o design a sysem o suppor 50 concurren users requesing maps. How many CPU cores are required o suppor his load? Soluion: Sep 1: Deermine he SpecRae of he proposed server using hp://www.spec.org/cpu2006/resuls/cin2006.hml, as described in Adjusing for CPU Speed Relaive Difference secion. IF here is no informaion abou he ype of CPU, assume he laes CPU is used. The spec raes were deermined as Spec_Per_CPU=30. Sep 2: Adjus for hroughpu. Since here is no informaion available abou frequency of ransacions, hroughpu is esimaed a 6 seconds hink ime and 1 second response ime. Sep 3: Selec he relevan Performance Benchmark repor (similar applicaion) from he Implemenaion Gallery. For inpu, use informaion in he Capaciy Planning Model Inpu, including spec rae and CPU service ime. Sep 4: Adjus for ransacion size. IF here is no informaion abou ransacion ype or size, assume he benchmarked CPU service ime. Sep 5: Esimae capaciy of he proposed sysem using Capaciy Calculaor RT 1.00 Sec Users 50.00 Users Think 6.00 Sec ST ArcSOC Zoom b 0.60 Sec Mbis/r b 1.81 Mbis/r SpecRae b PerCPU 13.425 15
SpecRae PerCPU 13.425 %CPU 95 % TH 25,714 r/hr CPU #CPU 4.48 cores Mbps 12.96 Mbps Soluion: Approximaely 5 CPU cores are required o suppor he given load. The curren servers are sold ypically wih 4 or 8 cores. Therefore, o accoun for fuure growh i is recommended o use 8-core server. 4.7 Example 2: Using faser CPU A hosing company charges $ 0.01 per map reques wih he performance level of service less han 1 second. The company has more poenial cusomers, however he curren sysem based on 4 core 3 GHz CPU is reaching he capaciy and users repor a degraded performance, wih a map response ime frequenly above 2 seconds. An IT manager considers upgrading he exising server o a newer one. He esimaes he oal cos of upgrading hardware is $30,000 How should he esimae he reurn on invesmen (ROI)? Soluion: Sep 1: Deermine he SpecRae of he exising and proposed (modeled) environmen using hp://www.spec.org/cpu2006/resuls/cin2006.hml as described in Adjusing for CPU Speed Relaive Difference secion. The spec raes were deermined as Spec_Per_CPU = 13.425 for he exising sysem and Spec_Per_CPU = 30 for he proposed sysem. Sep 2: Selec relevan Performance Benchmark repor and daa from Implemenaion Gallery. Ensure he benchmark and design applicaions are similar. Sep 3: Model he exising and proposed sysem using benchmark daa. Deermine he maximum hroughpu corresponding o conraced level of service (response ime=1 sec). Noe, we are no evaluaing he number of concurren users, bu he hroughpu. For his analysis, we can assume any hink ime and number of concurren users. For he purpose of analysis we will use he benchmark hink ime of 6 seconds. Also, he benchmark and he exising sysem are he same. If no boh sysem would need o be modeled. We inpu Spec_Per_CPU=30 in his workshee and analyze he resul. ThinkingTime Spec_Per_CPU Mbis_per_Tr SOCCPUCoun Benchmark (B) 6 13.425 1.81 4 Targe (T) or Modeled 6 30 4 16
We deermine he exising hroughpu corresponding o he required response ime by drawing a horizonal line (blue) from 1 sec Response Time and inersecing wih he Throughpu curve (blue). We can esimae he exising hroughpu a 19,000 map/hour. We repea he same seps (pink line) and esimae he new hroughpu a 38,000 map/hour a an average response ime less han 1 second for he argeed server Noe, for more precision, we can use Equaion 6: Throughpu as a funcion of CPU service ime and number of cores or benchmark daa o calculae he hroughpu. Since he sysem has o deliver 1 sec response ime, we should firs deermine he corresponding CPU uilizaion from benchmark daa as shown below (we can assume 60% ). 17
Sep 4: Conduc Cos/Benefi Analysis The new sysem can suppor an addiional 19,000 map/hour; he esimaed profi is $190/hour. Assuming he sysem will be consisenly uilized a 38,000 maps/hour for 10 hours a day, 5 day a week (10*5*4=200 hr/monh), his invesmen would yield 10*5*4*190=$38,000 poenial profi per monh. 4.8 Example 3: Using faser CPU o maximize hroughpu The same company from he previous Example considers renegoiaing he exising level of service agreemen. The managemen wans o evaluae wha would be an addiional monhly profi if he company did no have o guaranee he performance level of service (response ime under 1 sec)? Soluion: Follow Sep 1-2 from he previous example. Sep 3: Model he exising and proposed sysem. Deermine he maximum hroughpu. 18
New max hroughpu max = 47,000 Exising max hroughpu = 20,000 We can summarize he hroughpu and response ime for he exising and upgraded sysem in he following able: Max Throughpu Max User load (r/hr) Before 20,000 51 Afer 47,000 99 % Improvemen 130% 94% I can be esimaed ha by using faser CPU (30/13.425=2.2 imes faser) we improve boh performance and scalabiliy: Throughpu (scalabiliy) increased by 100% ; he curve exended User load (scalabiliy) increased by 94% Sep 4: Conduc Cos/Benefi Analysis Wih he new sysem we can suppor addiional 47,000-21,000=26,000 map/hour, wih he esimaed profi of $260/hour. Assuming he sysem will be used 10 hours a day, 5 day a week (10*5*4=200 hr/monh), his invesmen would yield 10*5*4*260=$52,000 profi per monh. 4.9 Example 4: Using more CPU cores 19
The same hosing company from he previous Example considers adding (doubling) CPU cores o he exising sysem. How will his improve performance and scalabiliy of he sysem? Howis reurn on invesmen esiamed? Soluion: Follow Sep 1-2 from previous Example. Sep 3: Model he exising and proposed sysem. Deermine he maximum hroughpu. In his case, we increase he CPU Coun o 8 in his workshee and analyze he daa. ThinkingTime Spec_Per_CPU Mbis_per_Tr SOCCPUCoun Benchmark (B) 6 13.425 1.81 4 Targe (T) or Modeled 6 13.425 8 Throughpu doubled Response ime no changed (up o 25 users) The following able summarizes he key saisics. Response Time (sec) Max Throughpu (a 23 users) (r/hr) Max User load 20
Before 0.73 21,000 51 Afer 0.73 43,000 92 % Improvemen 0% 100% 94% I can be esimaed ha adding more (doubling) CPU cores from 4 o 8: Response ime (performance) remains he same for he moderae user load (less han 25 users); he curve shifed o he righ. Throughpu (scalabiliy) increased by 100%; he curve exended. User load (scalabiliy) increased by 94% Sep 4: Conduc Cos/Benefi Analysis Wih he new sysem we can suppor addiional 43,000-21,000=22,000 map/hour, wih he esimaed profi of $220/hour. Assuming he sysem will be used 10 hours a day, 5 day a week (10*5*4=200 hr/monh), his invesmen would yield 10*5*4*220=$44,000 poenial profi per monh. 4.10 Example 5: Esimaing he value of uning The same hosing company from he previous Example hired a vendor o une heir exising sysem a he cos of $30,000. As a resul i was repor performance improved 50% (response ime 0.5 imes). Wha is he reurn on invesmen for he uning engagemen? Soluion: Follow Sep 1-2 from previous Example. Sep 3: Model he before and afer sysem. Deermine he maximum hroughpu. Based on he response ime reducion informaion, we can assume CPU service ime was reduced o 0.3 seconds. From he benchmark, we deermined he original sysem CPU service ime was equal o 0.6 seconds. We can hen calculae he corresponding capaciy of he sysem (4 core, 95% CPU uilizaion) using Equaion 6: Throughpu as a funcion of CPU service ime and number of cores: 3600 % CPU # CPU T T TH = STb 100 SpecRaePerCPU SpecRaePerCPU TH before uning =(3600*95*4)/(0.6*100)* 13.425/13.425= 22,800 r/hr TH afer uning =(3600*95*4)/(0.3*100)* 13.425/13.425= 45,600 r/hr T b As expeced, given all oher facors as consan, he reducion of CPU service ime by 50% yields double he hroughpu. Sep 4: Conduc Cos/Benefi Analysis Wih he new sysem we can suppor addiional 45,600-22,800=22,800 map/hour, wih he esimaed profi of $228/hour. Assuming he sysem will be used 10 hours a day, 5 day a week (10*5*4=200 hr/monh), his invesmen would yield 10*5*4*228=$45,600 poenial profi in one monh. 4.11 Example 6: Changing frequency of user operaions 21
The same hosing company from he previous Example observed ha he sysem was running a capaciy (95% average CPU uilizaion) and users repored performance degradaion. The IT manager analyzed he previous user workflow and anicipaed ha, users zoomed o an area of ineres hen analyzed he daa for 6 seconds before repeaing he operaion. He now esimaes he new and improved analyical ools allow users o reduce he analysis ime from 6 o 3 seconds. As a resul, users reques a new map every 3 seconds. The same hosing company is negoiaing new level of service agreemen based on he guaraneed maximum number of concurren users. Wha is he maximum number of users he company should agree o? Soluion: Follow Sep 1-2 from he previous example. Sep 3: Model he exising and proposed sysem. Deermine he maximum hroughpu. In his case, reduce Think Time from 6 seconds o 3 seconds. ThinkingTime Spec_Per_CPU Mbis_per_Tr SOCCPUCoun Benchmark (B) 6 13.425 1.81 4 Targe (T) or Modeled 3 13.425 4 22
Max Throughpu no changed Response ime no changed (up o 12 users) Max user load reduced from 51 o 29 The following able summarizes he key saisics. Response Time (sec) Max Throughpu Max User load (a 23 users) (r/hr) Before 0.77 21120 51 Afer 0.77 21144 29 % Improvemen 0% 0% <43%> I can be esimaed ha by increasing he frequency of requesing maps (shorening hink ime): Response ime (performance) remains he same for he moderae user load (less han 12 users). Maximum hroughpu (scalabiliy) no changed. User load decreased 43%, from 51 o29 users. Sep 4: Conduc Cos/Benefi Analysis Wih he new user workflow, he sysem can suppor a maximum of 29 users. 23
4.12 Example 7: Nework sizing - changing he size of he map display The same hosing company from he previous Example hoss an applicaion wih he map display size se a 800 x 600. Users have been requesing a larger map display size o improve usabiliy, e.g. 1200 x 1000. Currenly, he company pays $1000 a monh per 1 Mbps of bandwidh. How much exra should he company charge per reques? Soluion: Follow Sep 1-2 from he previous example. Sep 3: Model he exising and proposed sysem. We analyzed he published benchmarks for a similar applicaion. In addiion o hroughpu, we found ha a map display size of 1200 x 1000 requires 1.81 Mbis/r while an 800 x 600 size requires 0.6 Mbis/r. We can use he Capaciy Calculaor o conduc he analysis. The following able liss he curren requiremens: RT Users Think sec users sec ST ArcSOC Zoom b 0.60 sec Mbis/r b 0.6 Mbis/r SpecRae b PerCPU SpecRae PerCPU %CPU % TH #CPU Mbps 21,000 r/hr CPU cores 3.50 Mbps To calculae he new requiremens, we inpu Mbis/r=1.81. RT Users Think sec users sec ST ArcSOC Zoom b 0.60 sec Mbis/r b 1.81 Mbis/r SpecRae b PerCPU SpecRae PerCPU %CPU % TH #CPU 21,000 r/hr CPU cores 24
Mbps 10.56 Mbps Sep 4: Conduc he Cos/Benefi Analysis Assuming he sysem will be used 10 hours a day, 5 day a week, we esimae he number of requess per monh is 10*5*4 * 21,000 = 4,200,000 reques/monh. The addiional nework cos per monh is (10.56-3.50)*$1000= $7,060. The company should charge exra 7060/4,200,000=$0.0017 per reques. 4.13 Example 8: Nework sizing increasing hroughpu The same hosing company from he previous Example upgraded is server hardware and now suppors double he number of requess from 21,000 r/hr o 42,000 r/hr. The company currenly pays $1000 a monh per 1 Mbps. Wha is he addiional monhly nework fee? Soluion: Follow Sep 1-2 from he previous example. Sep 3: Model he exising and proposed sysem. We analyzed he published benchmarks for a similar applicaion. In addiion o hroughpu, we found ha a map display size of 1200 x 1000 requires 1.81 Mbis/r. We can use Capaciy Calculaor o conduc analysis. The following able liss he curren and new nework requiremens: RT Users Think sec users sec ST ArcSOC Zoom b 0.60 sec Mbis/r b 1.81 Mbis/r SpecRae b PerCPU SpecRae PerCPU %CPU % TH #CPU Mbps 21,000 r/hr CPU cores 10.5 Mbps RT sec Users users Think sec ST ArcSOC Zoom b 0.60 sec Mbis/r b 1.81 Mbis/r SpecRae b PerCPU SpecRae PerCPU 25
%CPU % TH #CPU Mbps 42,000 r/hr CPU cores 21.12 Mbps Sep 4: Conduc Cos/Benefi Analysis The company will have o pay (21.12-10.56)*$1000=$10,560 per monh. 5 Walk-hrough of Benchmark Resul Secions The following secions describe he informaion presened wihin a ypical benchmark repor. 5.1 Applicaion Archiecure This secion provides a brief overview of he esed applicaion. For pracical informaion on his opic, see ESRI Applicaion Archiecures. 5.2 Hardware and Sofware Configuraion This secion liss he esed hardware and sofware configuraion. For more informaion, see ESRI Applicaion Archiecures. The following is an example of a es configuraion. Figure 2: Tes Configuraion Example Diagram Key Web SOM SOC RDBMS File 5.3 Benchmark Resuls This is a key secion of he performance benchmark documen. I will repor performance and scalabiliy informaion. 26
5.3.1 Performance and Scalabiliy This secion conains key es resuls and summary informaion derived from he raw es values. The figure below shows an example of a key es resuls repor. Figure 3: Key Tes Resuls Sysem capaciy marker Defined by maximum hroughpu Lised under his char you will find a summary of he sysem capaciy. Capaciy of he sysem can be deermined as user load corresponding o one of he following crieria: a firs error maximum hroughpu desired response ime In ESRI benchmarks, capaciy is referred o as user load corresponding o maximum hroughpu. Below is an example of how capaciy informaion is repored: Maximum Throughpu (Transacions/Hour) A User Load Average Response Time (Seconds) 21,120 51 2.9 27
For more informaion on how o apply his informaion o your soluion, see Esimaing Capaciy Using Benchmark Daa. For deails on how o analyze load es resuls, see MSDN Library, Analyzing Load Tes Runs DS2009: ArcGIS Server Performance and Scalabiliy Tesing Mehodologies 5.3.2 Resource Uilizaion This secion includes key resource uilizaion ha can be used o validae es resuls and idenify soluion bolenecks. For example, in he case as shown below, he green line represens CPU uilizaion and demonsraes he sysem is CPU bound. I can be concluded ha by adding more CPU resources, he capaciy of his soluion could be expanded. Figure 4: Key Resource Uilizaion Couner Insance Caegory Compuer Color User Load _Toal Load Tes: Scenario ArcGIS Server % Processor Time _Toal Processor ArcGIS Server % Idle Time 0 C: Physical Disk ArcGIS Server Available MB - Memory ArcGIS Server Byes Received/Sec. Broadcom BCM5708C NeXreme II GigE Nework Inerface ArcGIS Server Range Min. Max. Avg. 100 1 51 26 100 1.76 100 58.7 100 96.8 100 99.7 10,000 5,909 6,378 6,093 100,000 8,989 84,503 26,905 28
For deails on how o analyze load es resuls, see MSDN Library, Analyzing Load Tes Runs DS2009: ArcGIS Server Performance and Scalabiliy Tesing Mehodologies 5.4 Capaciy Planning This secion provides he inpu for he capaciy planning model, ypically found as an Excel spreadshee wihin he benchmark package. 5.4.1 CPU SpecRae CPU SpecRae SpecRae is a sandardized meric allowing various sysems o be compared. See hp://spec.org for specific informaion and resuls. The following is an example of a repored Spec resuls: SpecRae/CPU = 13.425 Toal CPU Cores = 4 5.4.2 CPU Service Time CPU service ime is a measure of how many seconds, on average, were needed for he CPU processor o process one reques or ransacion. I is used as an inpu for sizing models. The following is an example of a service ime repored resul: 5.4.3 Transacion Size Web ArcGIS SOC/SOM Daabase.03 0.62 N/A: RDBMS no uilized The following is an example of nework bandwidh (?) repored resuls: Average map size: 244,275 byes 29
6 Helpful Resources DS2009: ArcGIS Server Performance and Scalabiliy Tesing Mehodologies hp://resources.esri.com/arcgisserver/apis/javascrip/arcgis/index.cfm?fa=mediagallerydeails&mediaid=6d73b2db-1422-2418-344143680a5154ba hp://proceedings.esri.com/library/userconf/devsummi09/papers/ performanceesing2009_devsummi_pp_v1.pdf Paerns & pracices: Performance Tesing Guidance for Web Applicaions hp://perfesingguide.codeplex.com/release/projecreleases.aspx?releaseid=6690#downloadid=17955 SpecRae CINT 2006 hp://www.spec.org/cpu2006/resuls/cin2006.hml MSDN, Chaper 2 Performance Modeling hp://msdn.microsof.com/en-us/library/ms998537.aspx Performance Tesing Guidance for Web Applicaions hp://msdn.microsof.com/en-us/library/bb924375.aspx MSDN, Chaper 16 Tesing.NET Applicaion Performance hp://msdn.microsof.com/en-us/library/ms998581.aspx Geing Sared wih Team Sysem Tesing Tools hp://msdn.microsof.com/en-us/library/ms243146.aspx Fiddler hp://www.fiddlerool.com/fiddler/version.asp Visual Sudio 2008 Professional Ediion (90-day rial) hp://www.microsof.com/downloads/deails.aspx?familyid=83c3a1ec-ed72-4a79-8961-25635db0192b&displaylang=en MSDN, Chaper 17 Load-Tesing Web Applicaions hp://msdn.microsof.com/en-us/library/bb924372.aspx Creaing a Web Tes hp://msdn.microsof.com/en-us/library/ms182538.aspx 30