1 Evaluating the impact of and Vitualization Softwae Technologies on the RIT Datacente Chistophe M Butle Vitual Infastuctue Administato Rocheste Institute of Technology s Datacente Contact: Sustainable Development Class Poject Mach 2, 2009
3 Executive Summay Ove the past decade the cost to powe and cool copoate Datacentes has gown damatically (Figue ES 1). The expenditue fo powe and cooling has begun to exceed the annual capital expenditue of seve hadwae. This gowth in powe and cooling equiements is theatening the sustainability of cuent datacente design. The cuent business pactices, seve pocessing capacity utilization ates, and cooling stategies suounding datacente opeations need to be eevaluated in ode to bing aging datacentes back into a sustainable business model. This study takes a look at the existing seve infastuctue of Rocheste Institute of Technology s datacente and povides ecommendations fo efficiency impovements. The ecommendations ae based on a detailed inventoy of seve hadwae in the RIT datacente and on obseved hadwae utilization ates. The potential impovements in efficiency ae based on two tends in the IT industy which ae to educe space and powe equiements using blade seves and to incease CPU pocessing utilization ates using seve vitualization. The expansion of computing sevices and esouces is a fundamental activity in datacentes. Ove time the isolation of these sevices fo secuity and availability easons has been accomplished though the opeating system bounday. This model of isolation has led to an exponential gowth in seve hadwae. To combat the enegy costs due to the gowth of IT sevices companies have utilized blade centes which place multiple individual seves into a single case whee powe, cooling, and netwok connectivity ae shaed. In ode to educe the amount of undeutilized seve hadwae companies have leveaged seve vitualization softwae which balances the load of individual opeating systems acoss a cluste of physical seves. Hadwae capacity can be added to meet new sevice demand without a disuption of sevice. Many existing datacentes have seen damatic eductions in opeating and capital expense due to the impoved efficiency of hadwae and the inceased utilization of that hadwae. Sun has published the outcome of multiple datacente consolidation pojects outlining the pocess and the esulting cost savings (Figue ES 2). RIT has the potential to save millions of dollas ove the next 5 to 10 yeas by stategically leveaging cost efficient technologies, scutinizing existing hadwae puchases fo pefomance/watt atings, and encouaging individual depatments to consolidate seve hadwae into highly utilized cental seve clustes.
4 Figue ES 1: Powe and Cooling in U.S. Maket (Geenits) Figue ES 2: Sun Consolidation Results (Geenits)
5 Evaluating the impact of and Vitualization Softwae Technologies on the RIT Datacente Chistophe M Butle Vitual Infastuctue Administato Rocheste Institute of Technology s Datacente Intoduction Mach 2, 2009 The Rocheste Institute of Technology (RIT) datacente houses 314 physical seves. These seves ae owned and opeated by multiple depatments within RIT. The Infomation Technology Sevices (ITS) depatment manages 130 of these seves. Vaious individual colleges and depatments ent ack space in the collocation aea which is a segment of the datacente whee individually secued acks ae povided to ITS customes fo an annual chage. Thee ae 80 seves in the collocation aea managed by vaious depatments. Anothe 104 seves in both the ITS opeated aea and the collocation aea ae opeated by vaious depatments and used fo eseach focused computing. The emaining 130 seves ae woned and opeated by the cental IT depatment (ITS). This study looks at the usage and chaacteistics of seves esiding in the datacente and makes ecommendations fo impoved efficiency and educed oveall costs. The seves ae gouped into ITS owned, collocation custome owned, and eseach computing focused. Ove the past two yeas the collocation aea was ceated and sold to RIT depatments. Many depatments took the oppotunity to leveage the lage cooling and battey backed powe (UPS) infastuctue of the datacente. The quick migation of seves into the datacente caused a damatic incease to the load on the datacente UPS. Today the load on the datacente UPS is appoaching 100% of its edundant capacity. Customes have additional seves that they plan to move into the collocation aea but ae being asked to wait until the powe constaint is dealt with. Thee ae two likely scenaios to addess the existing powe constaint which is theatening the continued gowth of the existing RIT datacente. The fist option will equie a lage UPS upgade to expand the powe capacity available fo the continued incease in quantity of seve hadwae. The second option is to damatically impove the efficiency of seve hadwae in the datacente while continuing to incease computing capacity fo yeas to come. The second option is moe difficult than the fist because it equies collaboation among the vaious colleges and divisions that own hadwae in the RIT datacente. It is my hope though this study to convince RIT s uppe management to cove the cost of upgading any depatment s hadwae in the datacente that is inefficiently consuming powe. This could povide the incentive fo all paties involved to wok togethe towad a common goal of educing excess waste of enegy, continually expand the total computing capacity of the datacente, and educe the oveall cost fo computing at RIT.
6 Tends of U.S. Datacente Enegy Consumption Accoding to a study poduced by Jonathan G. Koomey, Ph.D., based on data fom IDC, Intel, IBM, Dell, and othe souces, the aveage U.S. datacente has expeienced a 15% annual incease in powe consumption due to a gowth in the numbe of seves with only a small pat of that gowth fom inceased powe consumption pe unit (Jonathan G. Koomey, 2007). Data diectly fom IDC chats confim this gowth in powe consumption (Figue 1- Datacente total electic use by U.S. and wold datacentes, Figue 2 - U.S. Datacente Powe & Cooling Tend). A solution to the continued gowth in electic powe consumption is to impove utilization ates of hadwae and to efesh the seve hadwae when the pefomance/watt atio of olde seves falls too fa behind the pefomance/watt atio of the latest seve hadwae. Methodology This study audits the datacente seves and attempts to analyze the seve hadwae fo utilization ates and fo pefomance compaison though the use of Standad Pefomance Evaluation Copoation (SPEC.og) CPU benchmaks. Each model of seve in the datacente was given a pojected CPU pefomance ating based on benchmak esults published though the SPEC.og SPECint_ate2000 and SPECint_ate2006 benchmak suites. The aveage CPU utilization ate of ITS seves was gatheed though histoical CPU usage data in ITS s seve monitoing softwae (Hobbit) and applied as an aveage to the collocation custome seves. The computing cluste seve utilization ate was based on a wost case load given by the eseach computing system administato. These findings povided enough infomation to poject equied blade seve hadwae at a continuous 80% CPU utilization using VMwae s entepise vitualization softwae (ESX) based on cuent total CPU capacity in the datacente based on CPU intege pocessing ate benchmaks. Annual hadwae costs fo new HP blade seves wee based on cuent HP blade seve hadwae picing (Table 2 - HP Picing (Januay 2009)). Estimates fo existing seve costs wee based on lowest aveage seve picing pe focus goup; $2000/eseach compute node, $4000/seve in Collocation aea and ITS owned seves. The $4000/seve estimate is a low estimate when consideing the expense of lage multipocesso seves in the datacente whose initial cost exceeded $25,000/unit. Electic costs wee based on RIT s cuent $0.07/KWh as epoted by Catheine Ahen, diecto of engineeing sevices at RIT s facilities management depatment. The electic cost was used to calculate electicity costs fo seves based on ack powe monitoing and fom the datacente UPS epot on cuent input and output KVA (Table 4 - Rack Powe Obsevations). Powe and cooling estimates wee based fom vaious tools and obsevations. HP s blade system powe sizing tool (Table 3 - HP System Powe Sizing Tool v3.7.1) was used to calculate powe (watts) and themal load (BTU) of blade seves unning at 80%. Industy standad fomulas fo powe and cooling
7 calculations (Calculations and Costs) wee used to estimate changes in powe consumption of seves (watts) and the esulting cost to cool the themal load fom seves (BTUs). Scope The scope of this study is to evaluate the existing seves in the datacente and attempt to calculate 5 to 10 yea pojections fo powe, cooling, and CPU pefomance fo the continuation of cuent seve usage tends. The same 5 to 10 yea pojections wee used to look at potential powe, cooling, and CPU pefomance in the datacente if all seves wee vitualized and un on HP blade seves. Thee ae specific applications that equie special CPU achitectues that may not wok in a HP blade seve envionment. My assumption egading this exception is that it is less than 1% of existing sevices. This study consides factos like special gants and donations to be out of scope fo the pojections. In addition, the stoage hadwae and netwok infastuctue wee also consideed out of scope due to the enomity of combining multiple vitualization pojects into one study. Wheneve estimates wee used the esults eed on the consevative side to povide wost case benefit scenaios. Findings Calculations and pojections fo consolidation of hadwae in each aea of focus wee pefomed sepaately and also in combination. The same vaiables wee used acoss all aea pedictions except fo the initial data collected though obsevation. Reseach Computing Hadwae Reseach computing seves in the datacente have chaacteistics that set them apat fom the ITS and collocation custome seves. The eseach computing seves wee typically unning between 60% and 90% utilization which accoding to Sun (Figue 3 - Enegy Waste at Low Utilization) is within the ange of efficient usage of pocessing powe vesus electic use. The calculations and findings fo eseach computing ae included at the end of the appendix section. The cost benefit fo consolidating eseach computing hadwae on HP blade system hadwae did not povide a etun on investment in the fist 5 yeas of pojections. ITS s The ITS seves ae centally monitoed fo CPU usage and have detailed inventoy infomation available fo use by this study. The accuacy and amount of data available fo estimating CPU usage povided an aveage CPU utilization ate of 12%. That undeutilization of hadwae is an undelying cause fo excess powe consumption and inflated capital expense ove time. The ITS owned seve hadwae had a highe pecentage of olde hadwae compaed to eseach computing o collocation customes. Replacement and vitualization of this hadwae could yield significant eductions in powe and cooling costs ove the next 12 months and alleviate the need to eplace o expand the existing uninteupted powe supplies in the datacente. Collocation Custome s The seves in the collocation aea ae of simila natue to the ITS owned hadwae. The ange of CPU pefomance was identical to the ange found in ITS seves. Without diect access to monitoing data the CPU utilization ates wee estimated to be simila to ITS s seve hadwae.
8 Results The following gaphs wee geneated fom data collected and analyzed duing the study. The tabulated data fo the gaphs can be found in the appendix. Taget CPU Pocessing Utilization Rate of Datacente This gaph shows the pojected level of seve utilization esulting fom the migation of individual seves to HP blade seves unning VMwae s ESX vitualization softwae. The migation of existing seves will take appoximately 1 hou of sevice outage and will need to be scheduled with the sevice owne. Existing seves with excessive amounts of data will take up to 4 hous to migate. 90% Taget CPU Pocessing Utilization Rate of Datacente 80% 70% 60% 50% 40% 30% 20% 10% 0%
9 Reduction in the Numbe of Physical s due to Vitualization This gaph shows annual eduction of physical seves owned by ITS and collocation customes. Afte all standalone hadwae is eplaced in 2013 thee will appoximately 10 HP blade seves with enough memoy and CPU pocessing powe fo all poduction sevices in the datacente. Two additional blade seves may be equied fo test and development esouces. The HP blade seve chassis can hold 16 physical seves and has a 12 yea suppot life fom HP Reduction in the Numbe of Physical s due to Vitualization of Physical s Vitualization of DataCente excluding eseach computing Vitualization of ITS seves only Vitualization of Colocation Aea Customes only
10 15% Annual Gowth in numbe of Physical s with no Futhe Adoption of Vitualization This gaph shows the pojected numbe of physical seves in the datacente if thee is no conceted effot to impove esouce utilization. The 15% annual gowth is based on IDC data (Figue 2 - U.S. Datacente Powe & Cooling Tend). ITS is cuently using VMwae and Sun Solais vitualization technologies to educe the numbe of physical seves. 37% of the sevices managed by ITS ae un in vitual seves. Even with the highe level of hadwae utilization on the ITS seves unning vitualization softwae ITS has a total aveage utilization ate of 12% of existing CPU pocessing capacity. Impoving the utilization of CPU powe to a minimum of 60% would boast a damatic eduction in physical seves, capital expense, and electicity consumption (Figue 3 - Enegy Waste at Low Utilization) % Annual Gowth in numbe of Physical s with no Futhe Adoption of Vitualization Total # of physical seves in DataCente minus eseach clustes # of physical ITS seves # of physical Colocation Custome seves
11 20% Annual Incease in CPU Intege Pocessing Powe (Based on SPECint2006 Benchmaks) This gaph shows the equied CPU pocessing powe fo ITS and collocation custome sevices. The gowth ate assumes a 15% annual gowth in new sevices and a 5% annual gowth in existing sevices. The bottom ed line epesents the CPU pocessing powe of a single blade seve puchased in a paticula yea. The smooth blue line epesents the minimum equied CPU pocessing capacity based on cuent hadwae utilization and a 20% annual gowth ate. The geen line epesents 80% of the total available CPU pocessing powe of ITS and collocation custome hadwae afte the annual emoval of 4-5 yea old hadwae. The top puple line epesents the total available CPU pocessing powe fo ITS and collocation custome sevices. 50,000 45,000 40,000 20% Annual Incease in CPU Intege Pocessing Powe based on SPECint2006 Benchmaks Total CPU Int Capacity in DC (20% annual gowth ate; 15% fo new sevices, 5% fo existing)) CPU Intege Pocessing Rate 35,000 30,000 25,000 20,000 15,000 10,000 5, Usable CPU Intege pocessing in cente (keeping 20% fee) DataCente Requied CPU Intege Pocessing Load in ode to meet cuent and pojected demand DataCente Intege Pocessing Rate Annual Gowth pe
12 Requied Powe and Cooling Pojection fo all seves in Datacente (including eseach computing) These gaphs show the potential eduction in powe and cooling esulting in bette hadwae utilization. BTUs 1,400,000 1,200,000 1,000, , , , ,000 Requied Cooling Pojection fo all seves in DataCente (including eseach computing) Themal Load Pojection at 15% annual seve gowth without vitualization (BTUs) Themal Load on DataCente with migation to HP Cente(s) (BTUS) Watts 350, , , , , ,000 Requied Powe Pojection fo all seves in DataCente (including eseach computing) DataCente powe usage pojection without vitualization (Watts) (15% annual gowth) Total DataCente Powe Load fo seve hadwae with migation to HP blade seves(watts) Cuent Powe Load 50,
13 Annual Cost of Electicity Requied by ITS and Collocation Custome s in Datacente This gaph shows the cost fo the electicity equied to powe ITS and collocation custome seves. The 2008 data is based on obseved powe usage fom seve acks with powe stips that monito ampeage load. Fo seves without obsevable ampeage eadings the electicity was based on 50% of maximum non-edundant powe supply atings. $40, $35, $30, Annual Cost of Electicity Requied by ITS and Colocation Custome s in DataCente Electic Cost fo s in DataCente excluding eseach computing $25, $20, Electic Cost of s fo ITS seves only $15, $10, $5, Electic Cost of s fo Colocation Aea Customes only $
14 Annual Cost fo Cooling This gaph shows the cost fo electicity used to cool the datacente volume of ai (constant blue line) and the heat geneated by physical seves. The cost estimates ae based on RIT s cuent electicity cost of $.07/KWh and on powe equiements fo aveage Liebet ai chilling units. The existing Liebet chilles in RIT s datacente leveage passive cooling fom outside ai when outside tempeatues ae below 45 degees Fahenheit. Without passive cooling fom outside ai the chilles would equie 1KW of electicity fo evey ton of cooling equied. Thee is a 81.74% eduction in electicity used by the Liebet chilles when the outside tempeatue is below 45 degees. The estimated powe equied fo 1 ton of cooling capacity is 0.626KW based on aveage monthly minimum and maximum tempeatues in Rocheste. $40, $35, $30, $25, $20, $15, Annual Cost fo Cooling (BTUs) (12000BTU=1 ton of cooling; 1 ton of cooling=0.626 KW) Cost of latent cooling of Room Aea without the use of self contained ack cooling (BTU) (DC has 3136 squae feet) Cooling Cost fo s in DataCente excluding eseach computing Cooling Cost of s fo ITS seves only $10, $5, $ Cooling Cost of s fo Colocation Aea Customes only NORMALS YRS JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC ANN ROCHESTER Max ROCHESTER Min Wattage needed fo 1 Ton of cooling:
15 Annual Cost of HP s, chassis, netwok, and vitualization softwae (Excluding eseach computing) This gaph shows the annual expenditue equied fo adding HP blade seve hadwae, netwok connectivity, and VMwae s vitual datacente softwae. The annual cost afte 2013 dops damatically afte all olde stand alone seves ae emoved fom the datacente. The pojected costs include eplacement of blade seve hadwae that is 5 yeas old. The ongoing costs fom 2014 to 2019 ae fom the annual eplacement of 20% of the HP blade seve hadwae, fom hadwae maintenance, and fom softwae maintenance contacts. The HP System chassis has a suppoted life of 12 yeas and an intenal communication capacity of 5Tb/s which is adequate fo the next 10 yeas of datacente sevice gowth Annual Cost of HP s, chassis, netwok, and vitualization softwae (excluding eseach computing) Cost of hadwae, netwok, and vitualization softwae fo vitualization of ITS and Colocation Custome seves combined (cost savings though combined hadwae) Cost of Hadwae fo Complete vitualization of ITS s only Cost of Hadwae fo Complete vitualization of Colocation Custome s only
16 Annual Cost savings fo complete vitualization of existing seves in DataCente This gaph shows the pojected annual cost savings esulting fom 100% vitualization of poduction sevices owned by ITS and collocation customes. The cost savings include the annual eduction in cooling and powe equiements of seve hadwae. The cost of new HP blade seve hadwae, netwok connectivity, VMwae vitualization softwae, and maintenance contacts ae compaed to the cost of individual seve puchases at an aveage seve cost of $4000 (without ecuing maintenance costs) and an annual gowth of 15% in new seve hadwae (based on IDC data). In the next 12 months thee is a pojected net cost benefit of $8,000 though the emoval of any seve hadwae that is 5+ yeas old. A total net savings of $609,998 is pojected ove the next 5 yeas and a net savings of $3,254,756 is pojected ove the next 10 yeas. These estimates ae vey consevative numbes and do not include the potential cost savings fom not upgading the cuent datacente powe infastuctue, fom the eduction of netwok hadwae, o fom the consolidation of multiple stoage aea netwoks (SANs) in to a single vitualized stoage envionment. Annual Cost savings fo complete vitualization of existing seves in DataCente Total Savings due to Vitualization of all s in DataCente and switching fom oom cooling to individual ack cooling Total Savings due to Vitualization of s in DataCente Total Savings due to Vitualization of ITS s in DataCente Total Savings due to Vitualization of Colocation Custome s s in DataCente
17 Conclusions The potential cost savings to RIT fo inceasing utilization ates of seves in the datacente, deceasing powe consumption, and poviding a sustainable computing infastuctue is significant. I am estimating a complete etun on initial investment by the end of 12 months afte implementation when factoing in the cooling and electic costs. The estimated net savings ove 5 yeas is $609,998. The estimated net savings ove 10 yeas is $3,254,756. These estimates do not account fo futhe savings though the eduction in the numbe of netwok pots in use by ITS and collocation customes fom 527 (1Gb) pots to 4 (10GbE) pots. Futhe savings can be accomplished by changing the appoach to cooling in the datacente fom fou lage ton chilles to a single ack with self contained cooling. This educes the amount of latent heat enegy that must be cooled in a oom containing 3,136 squae feet of ack space and ambient ai down to a single ack with 8 squae feet of intenal aea. This eduction in ai volume equates to a eduction of 1,056,832 BTUs of latent heat enegy. (Imagine putting a lage window ai conditione in you attic at home and hoping that it passively cools the est of the house). Futhe study can be made to impove the accuacy of these pedictions. An audit of all seve hadwae puchases at RIT ove the past 5 yeas would povide a moe accuate gowth ate of seve hadwae and a moe accuate estimate of unit picing. The $4000 pe unit estimate in the study is assumed to be on the low side when factoing in the numbe of lage and expensive seves in the datacente. In summay I find the potential fo cost savings and eduction of envionmental impacts to waant an impovement in hadwae utilization ates in the datacente. I uge that all seve puchases ae evaluated on a pefomance pe watt basis. The cuent 5 yea lifespan expectancy of seve hadwae is costing RIT moe in electicity than it is saving in capital expense. A 5 to 6 yea old seve is appoximately 8 times slowe than a new seve at the same initial cost. I would like to see a maximum of 4 yeas of poduction use fo seve hadwae with the possible 5 th yea fo test and development use. At the end of 5 yeas the seve hadwae should be ecycled. I hope the findings of this epot help to justify immediate action. The potential fo savings is woth pusuing fo at least 1 to 2 yeas to veify the accuacy of the pojections. At wost the attempt to migate towad a 100% vitualized infastuctue can stave off the cost of eplacing the existing UPS hadwae.
18 Appendix Figues Figue 1- Datacente total electic use by U.S. and wold datacentes 2007) (Jonathan G. Koomey, Figue 2 - U.S. Datacente Powe & Cooling Tend (Geenits)
19 Figue 3 - Enegy Waste at Low Utilization (Geenits)
20 Figue 4 - CPU Intege Pocessing Rate Pe Watt 12 CPU Intege Pocessing Rate Pe Watt Intege ate pe watt Table 1- CPU Intege Pocessing Rate Pe Watt yea Intel Xeon Pocesso SPEC INT Benchmak Powe (W) Intege ate pe watt 1999 Pentium II Xeon Pentium III Xeon Xeon Xeon Xeon Xeon Xeon Dual-Coe Xeon Quad-Coe Xeon X Quad-Coe Xeon X Six-Coe Xeon L
21 Table 2 - HP Picing (Januay 2009) Qty Pat # Desciption Unit Pice Ext. Pice B22 HP BLc7000 CTO 3 IN LCD Encl $ 3, $ 3, B21 HP BLc W High Efficiency Powe Supply $ $ B21 HP BLc Encl Single Fan Option $ $ B21 HP BLc PH FIO Powe Module Opt $ $ Opt. 7FX c7000 Enclosue HW Supp $ $ 2, B21 HP System c-class 10Gb Long Range Small Fom-Facto Pluggable Option $ 2, $ 10, B21 HP BLc VC Flex-10 Enet Module Opt $ 9, $ 18, B21 HP BL495c G5 CTO $ 5, $ 78, UK068E HP 4y Nbd BL4xxc Sv Bld HW Suppot Complex 1 $ $ 2, Vmwae ESX licenses w/ 4y maintenance $ 6, $ 94, Total $ 211, Individual Cost $ 13, Table 3 - HP System Powe Sizing Tool v3.7.1 Powe/He at 80% Load Input Powe Input VA BTU Input Cuent Input Cuent Pe Cod Aiflow in CFM Aiflow in CMM Powe/He at 80% Load Input Powe Input VA BTU Input Cuent Input Cuent Pe Cod Aiflow in CFM Aiflow in CMM Powe/He at 80% Load Input Powe BTU Powe/He at 75% Load Input Powe BTU
22 Table 4 - Rack Powe Obsevations /Rack/Og Gouping Racks Numbe of seves (poweed On) Actual Powe Usage (Watts) Aveage Powe Usage (Watts) Reseach Computing I24, I26, I30, R14, R , Collocation Customes R1, R2, R3, R4, R5, R6, R7, R8b, R9a, R10, R11, R12, R , ITS Owned E25, E26, E27, E28, E31, E33, E34, E38, E39, E40, I25, I27, I28, I29, I38, I39, I , Totals , (with eseach clustes) (without eseach clustes) Table 5 - Summay of Initial data fo calculations Initial values fo collocation custome, ITS, eseach, total DC, and DC eseach DC with eseach: Resouces eplaced pe yea # of seves to emove Total CPU Intege pocessing powe of emoved seves Total Powe Reduction afte emoving seves DC without eseach: Resouces eplaced pe yea # of seves to emove Total CPU Intege pocessing powe of emoved seves Total Powe Reduction afte emoving seves ITS initial data: Resouces eplaced pe yea # of seves to emove Total CPU Intege pocessing powe of emoved seves Total Powe Reduction afte emoving seves
23 Colo custome initial data: Resouces eplaced pe yea # of seves to emove Total CPU Intege pocessing powe of emoved seves Total Powe Reduction afte emoving seves Reseach cluste initial data: Resouces eplaced pe yea # of seves to emove Total CPU Intege pocessing powe of emoved seves Total Powe Reduction afte emoving seves Calculations and Costs Powe and BTU calculations: Liebet Model VH267W 20-ton cooling units (o equivalent) ae used to cool the data cente. Each cooling unit uses a 5 hosepowe (Hp) fan to delive aiflow and consumes 3.73 kw fo aiflow demand and 16.7 kw fo themal demand (liebet.com). Total Heat Load = Room Aea BTU + Windows BTU + Total Occupant BTU + Equipment BTU + Lighting BTU Room Aea BTU = Length (m) x Width (m) x 337 Total Occupant BTU = Numbe of occupants x 400 Equipment BTU = Total wattage fo all equipment x 3.5 Lighting BTU = Total wattage fo all lighting x 4.25
24 Speadsheets used fo Gaphs Datacente including eseach computing:
25 Datacente excluding eseach computing:
26 ITS Calculations:
27 Collocation Custome Calculations:
28 Reseach Computing Calculations:
29 Woks Cited Geenits, A. (n.d.). Sun Fom Hot Spots to Cool Theads. Retieved Mach 2, 2009, fom Scidb: Jonathan G. Koomey, P. (2007). Estimating Total Powe Consumption be in the U.S. and the Wold.