Thermal-aware relocation of servers in green data centers

Transcription

1 Fronters of Informaton Technology & Electronc Engneerng engneerng.cae.cn; ISSN (prnt); ISSN (onlne) E-mal: Chaudhry et al. / Front Inform Technol Electron Eng n press 1 Thermal-aware relocaton of servers n green data centers Muhammad Tayyab CHAUDHRY 1, T.C. LING 1, S.A. HUSSAIN 2, Xnzhu LU 1 ( 1 Unversty of Malaya, Kuala Lumpur, 563 Malaysa) ( 2 COMSATS Insttute of IT, Lahore, 5 Pakstan) Emal: mtayyabch@yahoo.com; tchaw@um.edu.my; asadhussan@ctlahore.edu.pk; luxnzhu213@sswa.um.edu.my Receved May 13, 214; Revson accepted Oct. 28, 214; Crosschecked Abstract: Rse n nlet ar temperature ncreases the correspondng outlet ar temperature from the server. As an added effect of rse n nlet ar temperature, some actve servers may start exhalng ntensely hot ar to form a hotspot. Increase n hot ar temperature and occasonal hotspots are an added burden on the coolng mechansm and result n energy wastage n data centers. Ths may also result n falure of server hardware. Identfyng and comparng the thermal senstvty to nlet ar temperature for varous servers helps n the thermal-aware arrangement and locaton swtchng of servers to mnmze the coolng energy wastage. The peak outlet temperature among the relocated servers can be lowered and even be homogenzed to reduce the coolng load and chances of hotspots. Based upon mutual comparson of nlet temperature senstvty of heterogeneous servers, ths paper presents a proactve approach for thermal-aware relocaton of data center servers. The expermental results show a coolng energy savng by as much as 2.1 kwh, lowerng the chances of hotspots by over 77% and helps the establshment of green data centers. Key words: Servers, Green data center, Thermal-aware, Relocaton do:1.1631/fitee.1174 Document code: A CLC number: 1 Introducton Correspondng author Zhejang Unversty and Sprnger-Verlag Berln Hedelberg 215 Data centers around the world consume an enormous amount of electrc power each year. An average data center consumes the equvalent amount of electrcty as 25, homes n U.S.A. (Assure, 211). The cost of electrcty expendture exceeds the total captal expendture over the workng lfe of servers. Apart from computng, a major amount of electrcty s also consumed n coolng the servers. Ths s because a data center has a closed envronment and the electrcal power consumed by IT equpment s converted nto heat (GmbH) and an equal amount of power s needed to remove that heat and to mantan a proper workng envronment va coolng. Data centers must apply energy savng technques to go green as a large part of the electrcty s generated by burnng fossl fuels. Consderng the hot/cold asle rack arrangement over rased floor desgn of a data center (EPA, 27), the coolng cost can be as much as the computng cost n terms of electrcty used. Power Usage Effcency (PUE) s the rato of total electrcty usage by the data center to the electrcty used for computng. If the coolng nfrastructure conssts of mechancal chllers only (Lu et al., 212), then the PUE value may be equal to 2. unless power savng practces are adopted. Recent studes have shown a lttle decrease n data center PUE worldwde to 1.93 (Koomey, 211). The tradtonal approach of server consoldaton to save computng power may result n utlzaton of few servers to ther lmt. But the electrcty consumpton and the resultng heat dsspaton from these servers reach to maxmum n a small area of the data center. Rse n nlet ar temperature can further ncrease the outlet temperature to such a lmt that a hotspot s formed. A hotspot may trgger the otherwse dle coolng mech-

2 2 ansm to start coolng or t may prolong the coolng process for an already actve coolng mechansm. In both cases, the coolng s boosted for a larger area than that of the hotspot and more power s spent for coolng than consumed by the computng tasks nsde that hotspot. By avodng the chances of hotspots, the extra burden over coolng nfrastructure can be avoded and power can be saved. There are multple factors whch may combne to provde sutable condton for a hotspot. Among these factors s the physcal phenomenon of cold ar gettng warmer as t reaches the nlet of the servers mounted near the top of racks. Furthermore, some systems such as the legacy servers are less power effcent and thus dsspate more heat. It depends upon the processor model beng nstalled n the server. The processor s the most power consumng and thus the most heat dsspatng hardware equpment on the motherboard (EPA, 27). Legacy processor archtecture lacks the adaptve power usage capablty and consumes more power compared to modern processor (Huck, 211) and therefore dsspate more heat (Manuel Masero, 212). Servers whch consume comparatvely more power when dle are more prone to gve rse to hotspots than others. A server s consdered power effcent f t consumes comparatvely less power when dle and provdes more computng power per watt n terms of MHz per watt when actve. The chances of hotspot can be foreseen by analyzng the heat dsspaton of dfferent servers at varous locatons nsde a data center wth respect to nlet ar temperature. Ths s due to the fact that some servers may dsspate more heat at hgher nlet temperature whle some servers may not be that senstve, owng to the hardware archtecture. Based upon ths fact, f the physcal locaton of each server s determned accordng to the nlet temperature senstvty, the peak outlet temperature of the servers and the chances of hotspots can be reduced. For the servers that are already mounted n racks, the nlet temperature senstvty analyss can help n rearrangng the locatons of a set of servers. Hotspot avodance based relocaton of data center servers can be a part of capacty plannng or t can be done n parallel wth tradtonal coolng mechansm optmzaton based technques (Lee et al., 212). The server relocaton technque proposed n ths paper s a novel Chaudhry et al. / Front Inform Technol Electron Eng n press approach for mnmzng the chances of hotspots and greenng the data centers. The rest of the paper s arranged such that secton 2 descrbes the related lterature revew and secton 3 explans the energy model prelmnares. Secton 4 refers to the server relocaton algorthm whch s tested on expermental setup, explaned n secton 5. A comprehensve dscusson s provded n secton 6 to analyze the expermental results and the applcaton of the server relocaton algorthm. The paper ends wth recommendatons for server relocaton n secton 6.1 and a concluson n secton 7. 2 Related work A power proflng based thermal map predcton and equpment relocaton technque was reported by (Jonas et al., 21). Thermal map predcton was based on power profles of the server chasss. On the bass of the fact that every chasss makes contrbuton to the heat recrculaton of all the chasss n the data center, an equpment relocaton algorthm was proposed. However, t s practcally complex to access the heat recrculaton contrbuton coeffcent for each of the hundreds of chasss n a data center. These technques should only focus on the hotspot servers to decrease the complexty of mplementaton. If the servers wth hgh electrcty consumpton or the servers havng hgh utlzaton rate are placed at top of the racks where the nlet temperature s hgh, then dong so wll ncrease the outlet temperatures of these servers nstead of decreasng. If a power savng technque such as dskless bootng s used as proposed by (Che-Yuan et al., 21), then the servers wll dsspate even less heat f they are located n a way that the nlet temperature has mnmum effect on ncreasng the outlet temperatures of the servers. If the power consumpton profles of server are created so that the least power s used to execute a gven computng load and to ensure performance and proft as n (Kusc et al., 29), then the schedulng algorthm can save more power f the hotspots are avoded. Instead of usng a neural network to predct the outlet temperature, the thermal profles can be utlzed to predct the thermal map and chance of hotspots (Jonas et al., 27; Jonas et al., 21). Inlet

3 temperature varaton may lead to hotspot and cause an addtonal burden to the coolng mechansm. Data center energy effcency and power consumpton based schedulng technques can perform better f the computatonal workload s dstrbuted among servers on the bass of comparatvely lower nlet temperature preference (Tang et al., 27; Mukherjee et al., 29; Ahuja, 212). Smlarly the reducton n recrculaton of heat s more effectve f the servers are arranged accordng to ther senstvty to nlet temperature hke (Tang et al., 27). The research amng to ncrease the thermostat settng (Banerjee et al., 21; Banerjee et al., 211) for cold ar n data center or to model the thermal map (Qnghu et al., 26; Ahuja et al., 211) should consder the optmzaton of server locatons as a prerequste to mplementaton. Ths s also applcable to recent ASHRAE (ASHRAE-TC-9.9, 211) standards for enhanced nlet temperatures. The coeffcents of heat recrculaton and heat extracton for the data center servers (Qnghu et al., 26) are senstve to the nlet temperature ncrement and the value of coeffcents should not be affected by ths phenomenon. If servers are utlzed to the maxmum level through backfllng (Lzhe et al., 29; Wang et al., 29), then the chances of hotspots are ncreased wth the ncrease n nlet temperature. Therefore the nlet temperature should be consdered before backfllng or the servers should be relocated accordngly and then backflled to avod hotspots. Smlarly, the server consoldaton technques for mnmzng the number of actve servers should only choose those servers whch wll not cause hotspots. Ths s because the consoldated servers wll be at ther peak utlzaton all the tme (Corrad et al., 211). Task-temperature profles used for thermalaware workload schedulng should consder the effect of nlet temperature senstvty of the physcal servers upon the schedulng outcome n terms of the thermal map to be unexpected (Wang et al., 212). The thermal proflng based technques such as (Rodero et al., 212) and (Ivan Rodero et al., 21) cannot estmate and create the generc profles of all the homogenous servers whch are located at dfferent nlet ar temperature across data center. Hence there s a gap n research related to the thermalaware arrangement of data center server. Ths paper Chaudhry et al. / Front Inform Technol Electron Eng n press 3 presents a thermal-aware server locaton evaluaton and relocaton of vrtualzed data center servers to optmum locatons. Ths results n preventon of possble hotspots and coolng energy savng as well as the ncreased effectveness of thermal-aware schedulng technques. Coolng-aware workload placement wth performance constrants was proposed by (SansotteraandCremones, 211) n whch the rse n nlet temperature s due to heat recrculaton. The heat recrculaton was consdered to be due to the ar flow. The temperature of hot ar from each server was declared to be due to power consumpton accordng to computatonal workload on that server. Varous test case scenaros were analyzed by CFD smulatons wth dfferent power consumpton levels for the servers n order to profle each server for the heat recrculaton. Ths approach s based upon the pror work of (Qnghu et al., 26). These profles were used to evaluate the hghest possble thermostat settng, the lowest possble heat recrculaton and maxmze the performance. Ths approach leads to the utlzaton of each server accordng to the thermal-profle. The smulated scenaro of the data center has two CRAC unts at two opposte boundares of the data center hall. One of these CRAC unts was turned off so that hot ar would be removed less effcently from that regon and heat recrculaton may occur. In such a case, the servers whch have a hgh heat recrculaton mpact are always underutlzed. The servers consume up to 6% of peak power n the dle state as shown by our experments. Therefore t s not energy effcent to keep some servers dle or underutlzed. Instead, we propose to dentfy the servers whch are affected by heat recrculaton and dentfy the outlet temperature at varous utlzaton levels. Then, such servers can be relocated at other locatons nsde the data center so that these servers can be utlzed to maxmum wth comparatvely lower outlet temperature at the new locaton. In poneer work, (Qnghu et al., 26) proposed to create the heat recrculaton profles and heat-ext profles of the data center servers by usng varous power consumpton levels n CFD smulatons. These profles can be used to predct the thermal map of the data center, gven a power dstrbuton vector and heat recrculaton coeffcent matrx. Ths s a faster method for thermal predcton. How

4 4 ever, the CFD smulatons consume great tme n hours multples of number ten. Also, heterogeneous servers do not necessarly consume the same amount of power at the same level of CPU utlzaton and therefore do not have the same outlet temperatures despte the same nlet temperature. Ths makes the CFD based proflng approach prone to hardware related lmtatons that can only be verfed through experments upon real hardware as we show n ths paper. 3 Data center energy modelng prelmnares By the law of energy conservaton, the watts of electrcal power consumed are converted nto equvalent joules of thermal energy (GmbH). From now on the words power, electrcty and energy are used nterchangeably (where energy s consumed per unt of tme). If E computng s the electrcty consumed by a data center server then ths energy s converted to E joules as shown n Eq. (1). Chaudhry et al. / Front Inform Technol Electron Eng n press E computng =E joules (1) As explaned n (Moore et al., 25), power consumed by a water chlled computer room ar condtonng (CRAC) unts at HP labs s calculated wth reference to the cold ar set temperature and s called the coeffcent of performance (COP). The COP s the amount of work done w to remove heat Q as shown n Eq. (2). COP=Q/w. (2) The COP has a numerc value whch ncreases wth the ncrease n suppled cold ar temperature. The electrcal energy E coolng consumed to remove the heat dsspated by a server by supplyng the cold ar at a set temperature T receved can be wrtten as n Eq.(3). The Eq. (3) however dffers from (Moore et al., 25) as t consders the nlet ar temperature that each server s recevng. E computng =E joules/cop(t receved). (3) The cold ar gets hot when t travels towards servers from the perforated tles of a hollow floor and due to heat recrculaton. The hke n nlet ar temperature has a drect mpact on outlet ar temperature for each server. The servers near the top of each rack are the vctms of ths phenomenon. These servers wll dsspate more heat because the nlet ar temperature ncreases despte the fact that they mght not be utlzed at full. So the servers at the top of the racks put more burden on the coolng system than the servers near the floor, because of the rse n nlet ar temperature. The COP curve (Moore et al., 25) s unable to gve a soluton to ths. Therefore f the total electrcty consumpton E Total for runnng a server can be wrtten as n Eq. (4), Usng Eq. (3) E Total =E computng +E coolng, (4) E Total=E computng +{E joules /COP(T receved)}, (5) Usng Eq. (1) E Total=E computng +{E computng /COP(T receved)}. (6) The total energy consumpton of data center can be wrtten as n Eq. (7) E E E or E, ( ) n n computng Total Total computng 1 1 COP Treceved E (7) 1 1. ( ) (8) n Total Ecomputng 1 COP Treceved The total electrcal energy consumed by a data center as shown n Eq. (8) contans the electrcal energy consumed by physcal servers and the coolng system. Thus wth the knowledge of electrcty consumpton of the servers, the data center s coolng energy and therefore the total energy consumpton can be calculated. Ths paper proposes to calculate the coolng energy consumpton for each server based upon the nlet ar temperature at that server. Ths means that the COP value should not be taken at the set nlet ar temperature T set of CRAC. A useful fact s that the COP value rses wth the rse n

5 nlet ar temperature. So a server that s recevng the cold ar at a hgher temperature wll be responsble for a smaller share n total coolng energy consumpton accordng to Eq. (8). On the other hand, the servers havng hgh temperature of nlet ar T receved wll have a correspondng ncrease n the outlet ar temperature as shown n Eq. (9). Chaudhry et al. / Front Inform Technol Electron Eng n press 5 ΔT =T receved T set, (9) where ΔT s the ncrease n nlet temperature of server and causes the equvalent ncrease n outlet temperature of the server. The orgnal outlet temperature can be gven by the followng T outlet=t outlet(ncreased) ΔT. (1) The ncreased outlet temperature of a server due to ncrease n nlet temperature not only puts extra burden on the coolng mechansm but also may form hotspots. The formal effect s ndependent of workload on the server whle the latter occurs when the server s executng the workload. The servers wth hgher than set temperature of nlet ar T receved>t set means that the coolng energy s wasted for any server as shown n Eq. (11). ΔE T_nlet =E T_set E T_receved, (11) where E T_set s a subset of electrcty used to provde cold ar to server at temperature T set but t should have been E T_receved nstead. Ths s because T receved >T set and therefore E T_set >E T_receved. So ΔE T_nlet s the energy wasted for server. As a result the outlet temperature rses by ΔT degrees as per Eq. (9). Ths causes an equvalent energy to be wasted to cool the outlet ar of server that s extra hot by ΔT degrees. Therefore the total coolng energy wasted s the sum of coolng energy wasted and the extra coolng energy spent for all servers and can be gven by Eq. (12): n coolng _ wasted T _ nlet T _ nlet 1 n 2 ET. 1 E ( E E ) (12) The value of ΔE T_nlet wll be equal to or more than zero dependng upon the poston of server wth respect to floor. In ths paper, we propose to mnmze the energy wastage on coolng as gven by Eq. (12). The maxmum allowed nlet temperature can be represented by T max beyond whch ether the hotspot can occur or the server hardware may fal. We defne a problem statement for equpment relocaton as: If there exsts server such that T receved >T set Fnd server j where T j receved <T receved And T j outlet<t outlet And T outlet <T max And T j outlet<t max Subject to post relocaton condtons gven by: ( relocated ) Toutlet Toutlet And T j outlet(ncreased) <T outle And (T j receved T set )<(T receved T set ) And E coolng_wasted s mnmzed The energy wasted n coolng as shown n Eq. (11), can be mnmzed by equpment relocaton. Snce ths energy wastage s curable and therefore should not be ncluded n calculatng the total energy consumpton. So the Eq. (8) can be generalzed to n 1 ETotal Ecomputng 1 Ecoolng _ wasted. 1 COP( Tset ) (13) Usng Eq. (6) and Eq. (12), the value of E coolng_wasted can be solved. The ncrease n coolng energy wastage results n a rse n data center PUE. The calculaton of E coolng_wasted can be performed through the thermodynamcs model gven by (Qnghu et al., 26) whch requres the data on the blow rate of the server fans. The servers used for demonstraton (as shown n Table 1 n ths paper have dynamc fan rates and thus the calculaton of the E coolng_wasted becomes complcated. Instead, the coolng cost was calculated ndrectly through the approach proposed by (Moore et al., 25). Accordng to ths, the coolng cost s calculated wth reference to the COP of the set temperature of the CRAC unt. We have extended ths model for the calculaton of E coolng_wasted before applyng the relocaton algorthm. The energy wasted s calculated as a dfference between the coolng energy for the suppled ar temperature and the temperature of the cold ar receved by the servers. Ths s demonstrated n Eq. (14).

6 6 E m coolng _ wasted 2ET 1 m Ecomputng E computng 2*. COPT ( ) COPT ( ) 1 set receved Chaudhry et al. / Front Inform Technol Electron Eng n press (14) Ths paper proposes lowerng the coolng energy wastage by adjustng the locaton of the servers. Whle t may not be possble to totally elmnate E coolng_wasted a possblty s to normalze the coolng energy wastage due to ncrease n nlet temperature by lowerng the average outlet temperature of the affected server/s through relocaton. Consderng the same volume of ar at temperature T 1 degrees s heated to T 2 degrees, then the heat at temperature T 2 s greater than at T 1 (BBC, 214). Applyng the same concept on server outlets; f the temperature of the server outlet s lowered, then ths sgnfes the lowerng of heat. As from Eq. (1), f the server s relocated to a locaton wth comparatvely lower nlet temperature, then the amount of heat dsspated s lower because the temperature of the outlet s comparatvely lower at the new locaton. Thus, the coolng load s lowered. For the server whch s relocated to the regon of hgh nlet temperature as a locaton exchange, f the outlet temperature s lower than that of the prevous server at the same locaton, the overall coolng load of both servers s decreased. The more the homogeneous and comparatvely lower are the temperatures of the relocated servers, the lower s the coolng load. From Eq. (1), t can be nferred that the outlet temperature depends upon nlet temperature and the server utlzaton level. If the server utlzaton remans the same, a change n nlet temperature has a drect mpact upon outlet temperature. Ths allows predcton of the outlet temperature of server at current nlet temperature T receved wth respect to the nlet temperature of server j. Thus T predcted_outlet=(t outlet T receved) + T j receved. (15) The predcted outlet temperature of the servers and j can be used to evaluate the current locaton of each server for the possblty of hotspots. In ths paper, Eq. (15) s used to predct an array of outlet temperature values for any server. The next secton presents the expermental setup used to solve the relocaton problem. 4 Server Relocaton Algorthm Ths secton presents the server relocaton algorthm based upon the analyss and comparson of varous varables related to performance, power and temperature. These varables are represented by vectors as explaned n the prevous secton. Before procedng, t should be noted that the arthmatc and logcal operatons performed between two vectors are mplemented on the correspondng elements of vectors n tme sne the dentcal experments were run on all servers. There can be a lst of heterogeneous servers wth the experments performed and data gathered on atleast one member of each heterogeneous server type before relocaton. The relocaton algorthm can be appled on two servers at a tme. Therefore for the sake of smplcty, two heteregenous server types are consdered n the algorthm. These are named as server type A and server type B as shown n Table 1. One member from each type of servers s chosen for the mplementaton. The arthmetc and logcal operatons performed between a vector and a scalar are performed on each element of vector wth the same value of scalar. 1 Server_Relocaton (T A dle_nlet, T B dle_nlet, T A nlet, T B nlet, T A outlet, T B outlet, T max ) 2 { 3 If (T A dle_nlet<t B dle_nlet And (T A dle_nlet<t max And T B dle_nlet<t max )) 4 { 5 ΔT nlet = T B nlet T A nlet 6 ΔT outlet = T B outlet T A outlet 7 f (ΔT nlet > And ΔT outlet >) 8 { 9 f (ΔT outlet ΔT nlet ) 1 { 11 Calculate T B predcted_outlet through Eq.(15) 12 Calculate T A predcted_outlet through Eq.(15) 13 If (T A predcted_outlet T B outlet And T A predcted_outlet T B outlet) 14 { 15 swtch locatons of server type A and server type B 16 }

7 17 } 18 } 19 } 2 } The proposed algorthm requres the dle state nlet temperatures and other parameters of two servers at each run. The dle state nlet temperatures are requred to dentfy the dfference n nlet temperatures and that the nlet temperature s less than the vendor specfed maxmum temperature as shown n lstng (3). It s supposed that T max s same for all servers. Otherwse lstng (3) can have dfferent values for T max. The experments performed on the par of serves generate the data such as T A nlet, T B nlet, T A outlet and T B outlet. The algorthm proceeds further f server type A s located at an nlet temperature lower than server B. In lstng (7, 9) the checks are performed to confrm that the dfferences n nlet and outlet temperatures of the two servers are avalable and the outlet temperature dfference s larger than the nlet temperature dfference. Ths provdes an opportunty for predctng the outlet temperatures n lstng (11 12). If the predcted temperaure of server type A after relocaton s lower than the outlet temperature of server type Bbefore relocaton and the predcted temperature of server type B after relocaton s lower than the outlet temperature before relocaton then the algorthm suggests swtchng locatons of the servers. In the next secton, we explan the algorthm wth respect to experment sets n Fg. 2 where graphs shown are based upon the real data gathered from experments when the servers were placed at ntal locatons. The data s from three sources whch are: thermal sensors, smart power meters and the vrualzed hosts and s aggregated per host and per mnute to match the tme and hosts. 5 Expermental Setup The proposed approach was tested over a set of heterogeneous servers, runnng VMware ESX 5. (VMware, 29) hypervsor. We used vrtualzed servers (hosts) as the hypervsor can gve the detaled performance data and the frequency of server processor can be manpulated at run tme. Ths helps Chaudhry et al. / Front Inform Technol Electron Eng n press 7 n smulaton for varous processor frequences to smulate the routne at whch servers are actually used n data center. Servers were grouped accordng to ther processors models; Intel(R) Xeon(R) CPU E GHz and Intel(R) Xeon(R) CPU E GHz respectvely. The server groups were named Server Type A and Server Type B as shown n Table 1. The members of each server group are homogenous. For mplementaton, two servers; one of type A and other of type B were used. Server Type A B Table 1 Server types Processor Intel(R) Xeon(R) CPU E GHz Intel(R) Xeon(R) CPU E GHz To montor the nlet and outlet ar temperatures, external USB thermal sensors were used. The power consumpton of each host was measured by USB smart power meters. Each server has up to 8 vrtual machnes (VMs). Mcrosoft C# scrpt was used as the workload booster to manpulate the VM operatons. Each VM s runnng a CPU ntensve benchmark Prme95 ( Great Internet Mersenne Prme Search (GIMPS), 212) and s kept n suspended state. Each VM has a sngle vrtual CPU. Server type A was run dle for about 1 hours to prove the correlaton between nlet and outlet temperatures. Fg. 1 shows that the close correlaton occurs between the nlet temperature and outlet temperature of the prototype servers n dle state. Expermental results presented n a latter secton show that ths relatonshp holds when the servers s actve and ths s a basc matrx of evaluatng the post relocaton outlet temperatures from the set of servers nvolved. We performed three sets of experments nvolvng one server from group A and another from group B as shown n Table 2 at varous CPU frequences for server group A and B. Each experment set contans at least two servers. Snce the servers are heterogeneous and the dfference between the processor frequences s.8 GHz. Dynamc frequency scalng was used to vary the maxmum flps of the servers accordng to Table 2. The experment sets 2 and 3 have the servers runnng at same processor frequency and can approxmately represent the scenaros when the servers are runnng underutlzed.

8 Whle n set 1, both the servers undergo maxmum utlzaton. All the experment sets take 3 mnutes dle tme to start and then utlze the vrtualzed hosts (accordng to the frequency lmt of experment sets) for 3 mnutes and then brng the host to dle state to cool down to dle state temperature for 2 mnutes. Altogether, each experment takes approxmately 55 mnutes. The set temperature T set was at 21 Celsus. The ntal locaton of the servers was such that server type A was placed n a colder area and server type B was located n a hotter area. Table 3 shows the ntal condtons of the servers. On average, server type A uses less power when n dle state than server type B and ths causes the average outlet temperature to be hgher than the server type A n dle state as shown n Table 3. But server type A s recevng the nlet temperature at T set whch s lower than the nlet temoerature of server type B. Ths can be another reason of comparatvely lower outlet temperature of server type A. The COP for server type A s also lower than server type B accordng to nlet temperature receeved. Ths lead to a hypothess that the maxmum Chaudhry et al. / Front Inform Technol Electron Eng n press Inlet temperature varaton effect on outlet temperatures of servers Outlet temperature server type A (Celsus) Average nlet temperature (Celsus) Tme (Mnutes) Outlet temperature server type B (Celsus) Fg. 1 The correlaton between nlet temperature and outlet temperatures of type A and type B servers Table 2 The sets of experments performed. Each experment set nvolves both servers. Server type A has speeder processor than server type B. Experment Server type a processor Server type b proces- set frequency sor frequency GHz 1.86GHz GHz 1.86GHz 3 1. GHz 1. GHz temperature from hot ar outlet of server type A wll be lower than the maxmum temperature from hot ar outlet of server type B f maxmum power consumed by server type A s equal to the maxmum power consumed by server type B provded that server type A has a lower nlet temperature than server B. Table 3 The dle state statstcs of both servers. Server type B uses more electrcty n dle state Average dle Average dle Average nlet Server COP state power state outlet temperature type ( T receved ) consumpton temperature (Celsus) (watts) (Celsus) A B If both servers consume equal electrcty whle runnng dentcal workloads, then f the outlet temperature of server type B s greater than server type A then ths ndcates the mpact of nlet ar temperature. If server type B has a less powerful processor than server type A, then t wll be gvng less MHz per watt of power consumpton than server type A. If the processor of server type A has a hgher maxmum frequency and consumes less power n dle state and equal power at any level of processor utlzaton as compared to server B(ndcated by the outlet ar temperature), then t wll be worth predctng the outlet temperature of both servers after relocaton on the bass of nlet temperature dfference. In dle state of servers, suppose that vector T A dle_nlet and vector T B dle_nlet represent the nlet ar tme lapsed temperatures seres of server type A and Server type B respectvely. Assume that T A dle_nlet

9 and T B dle_nlet cover a reasonable tme to make nference. If P A dle and P B dle are the dle power consumpton vectors of server type A and server type B respectvely. Durng the experments mentoned n Table 2, the outlet temperatures of the servers were saved n vectors T A outlet and T B outlet for server types A and B respectvely. The nlet temperatures of the servers A and B were recorded n vectors T A nlet and T B nlet respectvely. Gven that T A dle_nlet<t B dle_nlet then T A max_outlet <T B max_outlet provded that P A max P B max. Where T A max_outlet s the maxmum outlet temperature vector from server type A when t consumes maxmum power P A max and T B max_outlet s the maxmum outlet temperature vector of server type B when t consumes maxmum power P B max. If ths hypothess s proved through experments gven n Table 2, then the next step wll lead to an algorthm for equpment relocaton.in the next secton, ths hypothess s tested and furher analyss s made to relocate the servers wth the objectve of lowerng the coolng load and to avod hotspots. 6 Expermental results and dscusson In ths secton, the expermental results are presented and dscussed wth respect to the server relocaton algorthm presented n the prevous secton. Lstng-1 of the algorthm can get the parameters from Fgs. 2.1 and 2.2 nwhch the servers under go the experment of maxmum workload and CPU utlzaton. As per Fgs , lstng-3 of the algorthm s true for both hosts as the dle state nlet temperature and the maxmum outlet temperature of server type A s always less than server type B whch ndcates that f the dfference n outlet temperature s dependent upon the dfference n nlet temperatures, then after swchng places, f the same experment s performed, then server type A wll have leser outlet temperature than server type B at same locaton. It makes the server type A a good canddate for swtchng locaton wh server B. In order to verfy that the nlet temperature and the outlet temperatures of both servers throughout the experment remaned such that the server type B always had the hgher nlet temperature and hgher outlet temperature than server type A, the operatons Chaudhry et al. / Front Inform Technol Electron Eng n press 9 Power (watts) Server Type A Exp. Set Server type A power usage (watts) Server type A outlet temperature (Celsus) Fg. 2.1 Server type A at maxmum power usage and maxmum outlet temperature Power (watts) Server Type B Exp. Set Server type B power(watts) usage Server type B outlet temperature (celsus) Fg. 2.2 Server type B at maxmum power usage and maxmum outlet temperature Inlet Temperatures Exp. Set Server type A nlet temperature (Celsus) Server type B nlet temperature (Celsus) Fg. 2.3 Inlet temperatures of Server type A and Server type B when Server type B s recevng hotter ar at nlet Dfference n nlet and outlet temperature of server type A and server type B Exp. Set Dfference n outlet temperatures (Celsus) Inlet temperatures dfference (Celsus) Fg. 2.4 The dfference between nlet temperatures s always hgher than the dfference between outlet temperatures for Server type A and Server type B at ntal locatons

10 1 of lstngs-5-6 are to be performed. If the dfference between the nlet temperatures ΔT nlet remans above zero (lstng-7) then t means that the outlet temperatures of both servers wll be such that the server type A wll have lower outlet temperature than server B. As shown n Fg. 3.4, the graph of outlet temperature dfference ΔT outlet remans around value 2 on y-axs. The dstance between ΔT outlet and ΔT nlet graphs shows how much more heat s dsspated from server type B than the nlet temperatures dfference. Ths dstance s qute sgnfcant n Fg when both servers run exp. set 2 & 3. We present the combned results of exp. set 2 & 3 n Fgs As per Eq. (1), all the electrcty s to be converted nto heat. Therefore, unless the server type A consumes more electrcty than server B, the outlet temperature of server type A wll reman less than server type B and the logcal comparson of lstng-7 wll be true. The power consumptons of both servers shown n Fgs show that the maxmum power consumpton of server type A s always less than or equal to the power consumpton of server type B throughout the experment. Hence the hypothess presented earler s proved. Ths s shown n Fg The bg hump n Fg. 2.4 of ΔT outlet graph s the sudden rse n outlet temperature of server B, whle the server type A took a whle to get heated. Ths may be due to the fact that server type B consumes more energy n dle state than server type A and when the exp. set s conducted over server B, the rate of rse n electrcy consumpton of server type B rses more sharply than server A durng a short nterval. CPU Usage (%) Server Type A Exp. Set Server type A CPU usage (%) Server type A CPU usage (MHz) CPU utlzaton and effectve Megahertz (MHz) of both servers are shown n Fgs Both servers under go maxmum utlzaton of CPU, Chaudhry et al. / Front Inform Technol Electron Eng n press Fg. 2.5 The maxmum utlzaton of Server type A CPU Usage (MHz) although they dffer n maxmum MHz. Before fnalzng the decsson to swtch locatons of both servers, the temperature predcon should be made. Ths s to fore see the effect of relocaton (lstngs ) and shown n Fgs The nlet temperature dfference ΔT nlet s added to the outlet of server type A to rase t and the same s subtracted from server type B outlet temperaure to lower t. CPU usage (%) Server Type B Exp. Set Server type B CPU usage (%) Server type B CPU usage (MHz) Fg. 2.6 Server type B has a lower maxmum frequency than Server B Outlet temperatures of server type A and server type B at current locaton Exp. Set Server type A outlet temperature (Celsus) Server type B outlet temperature (celsus) Dfference n outlet temperatures (Celsus) Fg. 2.7 The hump n outlet temperature dfference curve s due to a sudden rse n outlet temperature of server B Predcted temperatures for server type A and server type B Tme (mnute) Server type A pedcted outlet temperature (Celsus) Server type B predcted outlet temperature (Celsus) Predcted outlet temperature dfference (Celsus) Fg. 2.8 Predcted temperature of server type A and server type B after relocaton on the bass of nlet temperatures As shown n Fg. 2.9, the predcted temperature of server type A represented by T A predcted_outlet s less CPU usage (MHz) Temperature Dfference (Celsus) Temperature dfference (Celsus)

11 than the outlet temperature of server type B represented as T B outlet and the predcted outlet temperature of server B: T B predcted_outlet s also lower than T B outlet. Ths means that f the same experment s repeated after swtchng the locatons of both servers, then the server type B wll not only have a lower outlet temperature than the prevous locaton, but the server type A wll also dsspate less heat at the new locaton than server type B at the old locaton. Server type A and server type B current and predcted outlet temperatures Exp. Set Server Type A Outlet Server Type B Outlet Temperature (celsus) Server Type A Predcted Outlet Server Type B Predcted Outlet Fg. 2.9 The predcted temperatures wll lower the dfference between peak temperatures of both servers Power (watts) Server Type A Exp. Set 2 & Server type A powerc usage (watts) Server A nlet temperature (Celsus) Server A outlet temperature (Celsus) Chaudhry et al. / Front Inform Technol Electron Eng n press Fg. 2.1 Power consumpton and oulet temperature of Server-A runnng at lower frequency to match the processor of Server B Power (watts) Server Type B Exp. Set 2 & 3 Power Usage (watts) Server Type B Server Type B Inlet Server Type B Outlet Fg Power consumpton and oulet temperature of Server-B runnng at frequences equal to Server type A. The temperature s hgher than Server type A The logcal comparsons of lstng-13 are the post relocaton condtons that were presented earler n the problem statement and they gurantee that the relocaton operaton wll result n hotspot avodance and overall reducton n coolng load wth homogenous outlet temperatures of both servers. A sgnfcant dffernce between ΔT nlet and ΔT outlet shown n Fg whch shows a major mbalance n heat dsspaton when the servers are underutlzed. Server type A s beng under utlzed n exp. 2 & 3, but server type B s fully utlzed n exp. set 2 and under utlzed n exp. set 3. But server type B always consumes more energy than server type A and provdes less MHz per watt than server type A. Server type B dsspates more heat even when both servers run at almost smlar CPU frequences n exp. set 2& 3. Inlet Temperatures Exp. Set 2 & Server type A nlet temperature (Celsus) Server type B nlet temperature (Celsus) Fg Inlet temperatures of Server type A and Server type B when Server type B s recevng hotter ar at nlet Dfference n nlet and outlet temperature of server type A and server type B Exp. Set 2 & Outlet temperature dfference (Celsus) before movng Inlet temperature dfference (Celsus) before movng Fg The dfference between ar outlets and the ar nlets of Server type A and Server B. The formal dfference s much hgher than the latter. But the relocatn algorthm does not use ths CPU usage (%) Server type A Exp. Set 2 & Server type A CPU usage (%) Server type A CPU usage (MHz) Fg Server type A was run underutlzed to match the processor frequency of server B CPU usage (MHz)

12 12 CPU usage (%) Server type B Exp. Set 2 & Server type B CPU usage (% ) Server type B CPU usage (MHz) Now we move on to the verfcaton of post relocaton consderatons mentoned earler n problem statement. The same sets of expermens were performed over server type A and server type B after swtchng ther locaons. The results are shown n Fgs. 3.x where Fgs ndcae that server type B shows a reducton n outlet temperture whereas server type A outlet temperature s ncreased as compared to Fg wth the ncrease n nlet temperature. But the power consumpton of both servers follow the same trend as before relocaton (Fgs ) when exp. set 1 3 are repeated and so dd the nlet ar temperaures (Fgs. 2.3, 3.4 and 3.9). The hump of Fg. 2.4 for ΔT outlet s flatened n Fg. 3.3 showng a postve change n dfference between outlet temperatures. Ths shows that although the power consumpton of server type B shoots up n the start of exp. set just as n Fg. 2.2, but the rse n outlet temperature of server type B n Fg. 3.2 s balanced by the hgher rate of rse n oulet temperature of server type A n Fg Ths hump was responsble for an error n predcton of Fg. 2.8 for a short nterval of 4 mnutes. Chaudhry et al. / Front Inform Technol Electron Eng n press Fg Server type B runnng at ts full processor n Exp. Set 2 and at 6% of ts maxmum frequency Power (watts) Server Type A Exp. Set Server Type A Power Usage (watts) Server Type A Outlet Fg. 3.1 Server type A shows an ncrease n outlet temperature. But the power consumpton s same as before movng. CPU usage (MHz) Power (watts) Server Type B Exp. Set Server type B power usage (watts) Server type B outlet temperature (Celsus) Fg. 3.2 Server type B shows an decrease n outlet temperature. But the power consumpton s same as before movng Dfference n nlet and outlet temperatures of server type A and server type B n Exp. Set Actual outlet temperature dfference after relocaton (Celsus) Inlet temperature dfference after relocaton (Celsus) Predcted outlet temperature dfference (Celsus) Inlet temperatures dfference before relocaton (Celsus) Fg. 3.3 The predcted dfference n outlet temperatures was more accurate when the servers are runnng at maxmum utlzaton Fgs. 3.3 and 3.1 show that the predcted ΔT outlet curve follows closely to the curve of actual run. As shown n Fgs. 3.5 and 3.11, the predcted temperatures curves of T A predcted_outlet and T B predcted_outlet follow closely wth the actual outlet temperatures curves of T A outlet and T B outlet. Snce the predcted outlet ar temperature of server type A was less than that of server type B and that was a reason to relocate t, the fg. Fg.3.5 shows that the curve of T B predcted_outlet s more accurate than of T A predcted_outlet. The reason s that the T A predcted_outlet was calculated, based upon ΔT outlet whch depended upon the curve of T B outlet at the prevous locaton. Therefore accordng to lstng-13 of the algorthm, the servers could be relocated even f the T A predcted_outlet be equal to T B outlet before relocaton. Ths s proved by Fgs and The results of exp. sets 2 & 3 n Fgs and 3.12 prove the post relocaton scenaro. The sgnfcant dfference between the curves of ΔT outlet n Fg and of exp. set 3 n Fg. 3.1 s due to the fact that server type B dsspates more heat at low CPU utlzaton than server type A even when T A nlet T B nlet. The averaged results of the exper-

13 ments before and after relocaton are summarzed n Tables 4 6. The power consumpton of both servers s the same before and after the relocaton, but the change n outlet temperatures s notable. The calculaton for E coolng_wasted accordng to Eq. (14) shows that the server type B s wastng watts per mnute whle server type A s not wastng any energy due to havng a proper nlet temperature. As demonstrated n Table 5, after relocaton, server type A has a lesser ncrease n outlet temperature as compared to nlet temperature ncrease. Server type A and server type B nlet temperatures before and after relocaton Exp. Set Server type A nlet temperature after relocaton (Celsus) Server Type A nlet temperature before relocaton (Celsus) Server Type B nlet temperature after relocaton (Celsus) Server Type B nlet temperature before relocaton (Celsus) Fg. 3.4 The nlet temperatures at both locatons remaned almost the same Server type A and server type B outlet temperatures after relocaton and predcted temperatures before relocaton Exp. Set Server type A outlet temperature (Celsus) Server type B outlet temperature (Celsus) Server type A predcted outlet temperature (Celsus) Server type B predcted outlet temperature (Celsus) Fg. 3.5 The actual temperatures of the severs were wthn the predcted temperatures range Server type A and server type B outlet temperatures before and after relocaton Exp. Set Server type A outlet temperature (Celsus) Server type B outlet temperature (Celsus) Server type A outlet temperature (Celsus) Server type B outlet temperature (celsus) Fg. 3.6 The outlet temperatures of both server type are more homogenous after relocaton Chaudhry et al. / Front Inform Technol Electron Eng n press 13 Power (watts) Server Type A Exp. Set 2 & Tme (mnute) Server type a power usage (watts) Server type a nlet temperature (Celsus) Server type a outlet temperature after relocaton (Celsus) Fg. 3.7 The power consumpton of server type A was the same as before relocaton but the outlet temperature s hgher after relocaton Power (watts) Server type B Exp. Set 2 & Tme (mnute) Server type B power usage (watts) Server type B nlet temperature (Celsus) Server type B outlet temperature after relocaton (Celsus) Fg. 3.8 The power consumpton of server type B was the same as before relocaton but the outlet temperature s lower after relocaton Server type A and server type B nlet temperatures before and after relocaton Exp. Set 2 & Server type a nlet temperature after relocaton (Celsus) Server type a nlet temperature before relocaon (Celsus) Server type B nlet temperature after relocaton (Celsus) Server type B nlet temperature before relocaon (Celsus) Fg. 3.9 The nlet temperature remaned the same at both locaton as before relocaton Temperature Dfference (Celsus) Dfference n nlet and outlet temperatures of server type A and server type B Exp. Set 2 & 3 Outlet temperature dfference (Celsus) Inlet temperatue dfference (Celsus) Predcted outlet temperature dfference (Celsus) Fg. 3.1 The outlet temperatures of both servers are more homogenous after relocaton. The fall n curve of exp. set 3 s due to server type B dsspatng more heat than server type A even after relocaton

14 14 Therefore server type A compensated for ncrease n nlet temperature and therefore the E coolng_wasted s reduced to half for server type A by as much as 11 watts. Server type B has no E coolng_wasted due to lower nlet temperature. Therefore the savng n E coolng_wasted for server type B s from 22 to 24 watts as compared to Table 4. So overall the relocaton process saves over 5% of the coolng energy for the relocated servers, whch s Chaudhry et al. / Front Inform Technol Electron Eng n press Table 4 Summary of the expermental results before relocaton. Average computng power con- /COP(T E computng Experment Average Average E Total energy computng / E Server set before Inlet Outlet set ) coolng_wasted consumed COP(T type sumed (watts) per receved ) (watts) per (watts) per relocaton (Celsus) (Celsus) (watts) per mnute mnute (watts) per mnute mnute mnute A B A 2 & B 2 & Table 5 Summary of expermental results after relocaton wth calculatons of decrease n coolng burden. Avg. computng power temperatures before temperatures before Dfference n nlet Dfference n outlet Exp. set Avg. nlet temperature (Cel- temperature savng (watts Avg. outlet E coolng_wasted Server after relocatosus) (Celsus) per mnute) consumed and after relocaton and after relocaton (watts) (Celsus) (Celsus) A B A 2 & B 2 & Table 6 Decrease n occurrence of hotspots after relocaton. Exp. set Dfference between avg. outlet temperature before relocaton (celsus) Dfference between avg. outlet temperature after relocaton (celsus) Percentage less chances of hotspot after relocaton % 2 & % Server type A and server type B outlet temperatures after relocaton and predcted temperatures before relocaton Exp. Set 2 & 3 3 Server type A outlet temperature after relocaton (Celsus) Server type A predcted outlet temperature (Celsus) Server type B outlet temperature after relocaton (Celsus) Server type B predcted outlet temperature (Celsus) Fg The actual maxmum temperatures of the severs were wthn the predcted temperatures range Server type A and server type B outlet temperatures before and after relocaton Exp. Set 2 & 3 3 Server type A outlet temperature after relocaton (Celsus) Server type B outlet temperature after relocaton (Celsus) Server type A outlet temperature before relocaton (Celsus) Server type B outlet temperature before relocaton (Celsus) Fg The outlet temperatures of both server type are more homogenous after relocaton over 2.1 kwh each for the workng lfe of the servers. The relocaton of servers brngs homogenety n outlet temperatures whch reduces the chances of hotspots by up to 77.% as shown n Table 6. Ths can also be regarded as an mprovement n coolng energy wastage. The proactve approach proposed n ths paper saves the coolng energy that otherwse would be wasted on coolng a hotspot regon. If the relocaton algorthm s performed on the

15 server sets after relocaton, t wll not predct favorable outlet temperatures from the servers. Hence the algorthm performs well n reducng the outlet temperature and chances of hotspots once mplemented. The total energy consumed after relocaton s calculated and compared n Table 7. Snce there s no major change n computng energy consumed before and after the relocaton, therefore the coolng energy calculaton on the bass of COP (Moore et al., 25) wll not show the lowerng of coolng burden due to relocaton and/or decrease n outlet temperature of the servers. Ths s not only a lmtaton of the COP based calculaton but also s challengng to prove through thermodynamcs laws (Moore et al., 25) (Qnghu et al., 26). Ths s because the thermodynamcs laws are appled on the dfference between the outlet and nlet temperatures for heat calculaton and not on the ntensty of the temperature. Dervng new thermal laws or thermal engneerng equatons for heat calculaton s out of the scope of ths paper. Therefore the coolng energy savngs calculated n Table 6 are subtracted from the total energy consumed after relocaton to mark the benefts of comparatvely lower outlet temperature of the relocated server and the homogenety of the outlet temperatures of the relocated servers. Ths s demonstrated n Table Recommendatons for Server Relocaton Based upon the experments and results, we present the best practces for server relocaton nsde data centers. These recommendatons wll help the data center managers to dentfy, analyze, predct and perform a relocaton to save energy and to mnmze coolng energy wastage. The chances of legtmate relocaton requrement are hgher between a set of servers where a subset of the server has hgher nlet and outlet Server type Experment set Chaudhry et al. / Front Inform Technol Electron Eng n press 15 temperatures than another subset when all the servers are dle. If the server subsets have heterogeneous processors, then check the dle energy spent by the servers wth hotter outlet s hgher than the other subset. Ths wll add to the chance of relocaton as the hgher outlet temperature at dle state may gve rse to hotspot when servers are utlzed. As a next step, the server subsets should be marked and put to expermental test load whch boost the utlzaton of the servers CPU to maxmum and other underutlzed levels. Ths step can be skpped f the daly usage of servers CPU s avalable coverng a reasonable tme. But ths step s necessary f the servers are to be mounted for the frst tme. Data centers seldom keep the per mnute performance records of thousands of servers and keep the aggregated records nstead. Therefore, t s better to perform the exp. sets when there s an ndcaton of nlet/outlet temperature varance. If there s more than one server n each subset then the relocaton algorthm should be appled between all the combnatons of pared servers by takng one server from each subset. The relocaton algorthm gves the predcted temperatures of the par of servers. Ths can reduce the complexty of comparng servers. Each server should be dentfed wth the hghest predcted change n nlet and outlet temperatures. To make server pars, a good ndcator s that both servers use the same amount of maxmum electrcty. Relocaton s more favorable f a small change n nlet temperature can brng more change n outlet temperature. The rato of CPU MHz and watts consumed s a supportng value for the predcted temperatures. A server havng a low value of CPU MHz/watt wll dsspate more heat than another server wth hgher CPU MHz/watt value f the Table 7 Comparson of total energy consumpton before and after relocaton. Avg. computng power consumed (watts) E computng / COP(T set ) (watts) per mnute E coolngwasted (watts) per mnute E coolngwasted savng (watts per mnute) Total energy consumed after relocaton (watts) per mnute Total energy consumed before relocaton (watts) per mnute A B A 2 & B 2 &

16 16 maxmum power usage of both servers s equal to each other. It should be noted that at hgher nlet temperature, the outlet temperatures rses at hgher rate than at colder nlet temperature for the same server. The outlet temperatures of the relocated par of servers should be more homogenous and the post relocaton condtons defned earler n the problem statement should be satsfed. 7 Conclusons In ths paper we presented an energy model to represent the coolng energy wastage by nlet temperature varatons. The rse n nlet temperature can lead to hotspot causng ncreased outlet temperature of the data center servers. Ths ncreases the PUE of the data center due to energy wastage n coolng. Ths s hghlghted through the data center energy modelng presented n ths paper. The server relocaton algorthm can successfully optmze the locaton of each server to lower the extra burden on the coolng mechansm. The proposed approach can lower the chances of hotspots and mproves the coolng energy wastage by over 77%, lowers the coolng load through thermal-aware server relocaton leads to energy savng by 2.1 kwh throughout the servce tme span of relocated servers and thus helps n establshng the green data centers. In short, the partcular contrbutons of ths paper are: An energy model was presented to explan the effect of rse n nlet temperature of each server and the effect of ths upon total power consumpton of the data center. A proactve algorthm for server relocaton s presented to: Avod hotspots Lower the peak temperature of hot ar from the outlets of the relocated server set. Homogenze the outlet temperatures of the set of relocated servers These wll result n lower coolng load, avodance of hotspots, ensurng equpment safety and help n mantanng green data centers. Recommendatons or best practces for server relocaton are presented whch wll help the data center managers to dentfy, analyze and perform a Chaudhry et al. / Front Inform Technol Electron Eng n press relocaton to save power and to mnmze coolng power wastage. Conflct of Interest The authors have no conflct of nterest and all authors agreed on submsson and publcaton of ths manuscrpt n Journal of Zhejang Unversty- SCIENCE C (Computers & Electroncs). References Ahuja, N., 212. Datacenter power savngs through hgh ambent datacenter operaton: CFD modelng study. Proc. 28th Annual IEEE Semconductor Thermal Measurement and Management Symp., p [do:1.119/ STHERM ] Ahuja, N., Rego, C., Ahuja, S., et al., 211. Data center effcency wth hgher ambent temperatures and optmzed coolng control. Proc. 27th Annual IEEE Semconductor Thermal Measurement and Management Symp., p [do:1.119/stherm ] ASHRAE-TC Thermal Gudelnes for Data Processng Envronments Expanded Data Center Classes and Usage Gudance. Avalable from Assure, P., 211. Combnng PUE wth Other Energy Effcency Metrcs Avalable from s.pdf Banerjee, A., Mukherjee, T., Varsamopoulos, G., et al., 21. Coolng-aware and thermal-aware workload placement for green HPC data centers. Int. Green Computng Conf., p [do:1.119/greencomp ] Banerjee, A., Mukherjee, T., Varsamopoulos, G., et al., 211. Integratng coolng awareness wth thermal aware workload placement for HPC data centers. Sustan. Comput. Inform. Syst., 1(2):1-15. [do:1.116/j.suscom ] BBC, 214. Energy Transfer and Storage. Avalable from cty_forces/energy_transfer_storage/revson/1/ Corrad, A., Fanell, M., Foschn, L., 211. Increasng cloud power effcency through consoldaton technques. Proc. IEEE Symp. on Computers and Communcatons, p [do:1.119/iscc ] GmbH, L.D., Heat Heat as a form of energy Convertng electrcal energy nto heat (pp. 1). Germany: LD DIDACTIC GmbH, Leyboldstr. 1, D-5354 Hürth, Germany. Huck, S., 211. Measurng Processor Power TDP vs. ACP. Avalable from whte-paper/resources-xeon-measurng-processor-powerpaper.pdf Jonas, M., Varsamopoulos, G., Gupta, S.K.S., 27. On developng a fast, cost-effectve and non-nvasve method to derve data center thermal maps. Proc. IEEE Int. Conf.

17 on Cluster Computng, p [do:1.119/ CLUSTR ] Jonas, M., Varsamopoulos, G., Gupta, S.K.S., 21. Nonnvasve thermal modelng technques usng ambent sensors for greenng data centers. Proc. 39th Int. Conf. on Parallel Processng Workshops, p [do:1. 119/ICPPW.21.67] Koomey, J., 211. Growth n Data Center Electrcty Use 25 to 21. Analytcs Press, Oakland, CA. Kusc, D., Kephart, J.O., Hanson, J.E., et al., 29. Power and performance management of vrtualzed computng envronments va lookahead control. Cluster Comput., 12(1):1-15. [do:1.17/s y] Lee, E.K., Kulkarn, I., Pompl, D., et al., 212. Proactve thermal management n green datacenters. J. Supercomput., 6(2): [do:1.17/s ] Lu, Z., Chen, Y., Bash, C., et al., 212. Renewable and coolng aware workload management for sustanable data centers. ACM SIGMETRICS Perform. Eval. Rev., (1): [do:1.1145/ ] Masero, M., 212. CPU Charts 212: 86 Processors from AMD and Intel, Tested. Tom s Hardware. Mersenne Research, Inc., 212. Great Internet Mersenne Prme Search (GIMPS), 212. Avalable from Moore, J., Chase, J., Ranganathan, P., et al., 25. Makng schedulng cool : temperature-aware workload placement n data centers. Proc. USENIX Annual Techncal Conf., p Mukherjee, T., Banerjee, A., Varsamopoulos, G., et al., 29. Spato-temporal thermal-aware job schedulng to mnmze energy consumpton n vrtualzed heterogeneous data centers. Comput. Netw., 53(17): [do:1. 116/j.comnet ] Rodero, I., Lee, E.K., Pompl, D., et al., 21. Towards energy-effcent reactve thermal management n nstrumented datacenters. Proc. 11th IEEE/ACM Int. Conf. on Grd Computng, 21, p.1-8. [do:1.119/grid ] Rodero, I., Vswanathan, H., Lee, E.K., et al., 212. Energyeffcent thermal-aware autonomc management of vrtualzed HPC cloud nfrastructure. J. Grd Comput., 1(3): [do:1.17/s ] Sansottera, A., Cremones, P., 211. Coolng-aware workload placement wth performance constrants. Perform. Eval., 68(11): [do:1.116/j.peva ] Tang, Q., Mukherjee, T., Gupta, S.K.S., et al., 26. Sensorbased fast thermal evaluaton model for energy effcent hgh-performance datacenters. Proc. 4th Int. Conf. on Intellgent Sensng and Informaton Processng, p [do:1.119/icisip ] Tang, Q., Gupta, S.K.S., Varsamopoulos, G., 27. Thermalaware task schedulng for data centers through mnmzng heat recrculaton. Proc. IEEE Int. Conf. on Cluster Computng, p [do:1.119/clustr ] Chaudhry et al. / Front Inform Technol Electron Eng n press 17 Tu, C.Y., Kuo, W.C., Teng, W.H., et al., 21. A poweraware cloud archtecture wth smart meterng. Proc. 39th Int. Conf. on Parallel Processng Workshops, p [do:1.119/icppw.21.73] U.S. Envronmental Protecton Agency, 27. Report to Congress on Server and Data Center Energy Effcency Publc Law Avalable from gov/a/partners/prod_development/downloads/epa_dat acenter_report_congress_fnal1.pdf? f VMware, 29. VMware vsphere Bascs ESX 5. vcenter Server 5. (n Chnese). Avalable from vmware.com/vsphere-5/topc/com.vmware.icbase/pdf/ vsphere-esx-vcenter-server-5-bascs-gude.pdf Wang, L., von Laszewsk, G., Dayal, J. et al., 29. Thermal aware workload schedulng wth backfllng for green data centers. Proc. IEEE 28th Int. Performance Computng and Communcatons Conf., p [do:1. 119/PCCC ] Wang, L., von Laszewsk, G., Dayal, J., et al., 29. Towards thermal aware workload schedulng n a data center. Proc. 1th Int. Symp. on Pervasve Systems, Algorthms, and Networks, p [do:1.119/i-span.29.22] Wang, L., Khan, S.U., Dayal, J., 212. Thermal aware workload placement wth task-temperature profles n a data center. J. Supercomput., 61(3): [do:1.17/ s z]