Optimizing Data Centres Operation to Provide Ancillary Services On-demand Marcel Antal, Claudia Pop, Dan Valea, Tudor Cioara, Ionut Anghel, Ioan Salomie Technical University of Cluj-Napoca, Cluj-Napoca, Romania {marcel.antal, claudia.pop, dan.valea, tudor.cioara, ionut.anghel, ioan.salomie}@cs.utcluj.ro Abstract. In this paper a methodology for optimizing Data Centres (DCs) operation allowing them to provide various types of Ancillary Services on-demand is proposed. Energy flexibility models have been defined for hardware devices inside DCs aiming at optimizing energy demand profile by means of load time shifting, alternative usage of non-electrical cooling devices (e.g. thermal storage) or charging/discharging the electrical storage devices. As result DCs are able to shape their energy demand to provide additional load following reserve for large un-forecasted wind ramps, shed or shift energy demand over time to avoid an coincidental peak load and feed back in the grid the energy produced by turning on their backup fossil fuelled generators to maintain (local) reactive power balance under normal conditions. Experiments via numerical simulations based on real world traces of DC operation highlight the methodology potential for optimizing DC energy consumption to provide Ancillary Services. Keywords: Data Centre, Energy Consumption Optimization, Ancillary Services, Energy Flexibility, Demand Shifting 1 Introduction The Data Centre (DC) services business is blooming but, as it is usually the case, this is only one side of the story: the growing demand of their services increases their demand on energy resources, which directly translates to increasingly higher operational costs, not to mention the detrimental impact to the environment and, as such, society as a whole. Besides, the significant economic and environmental impact, the annual increasing energy demand of the DC poses another severe risk: the risk of supply shortage and instability of the electricity network. This may cause exponentially increasing side effects. On one hand to the local economy, which may suffer accidental black-outs, and on the other hand to the normal operation of the DC itself, as it is expected to provide continuous operation and guaranteed availability ranging from 99.671% (Tier 1 DC) to 99.995% (Tier 4 DC). All these factors are putting DC business in a risky position and creating higher pressure on the DC administrators on cutting down the energy demand and bills. Springer International Publishing Switzerland 2015
At the same time with the latest developments in the areas of digital technologies and renewable energy production the Smart Grid concept has emerged. It allows for two-way communication between the utility and its customers, the sensing along the distribution lines and combines traditional brown energy sources with green energy sources such as photovoltaic panels, wind turbines, geo-thermal power plants, etc. However, the main problem of this grid is that it cannot store energy, thus forcing the energy producers to shed their generation to match their customers energy demand. Variations of the production, either surplus or deficit can cause serious problems in the grid, leading to the overload of energy components and culminating with power outage or service disruptions. To make the problem more difficult to solve, the integration of renewables into the grid has added a level of uncertainty because of the intermittent and unpredictable nature of green energy generation. The lack of storage forces the grid operators to continuously deploy fast-reacting power reserves to maintain grid balance. To help solving this problem, utility companies developed Ancillary Services programs that allow them to ensure the energy balance in their grids, including mechanisms such spinning and non-spinning reserves or regulation. At the beginning of a billing period, a regulation signal is sent to every customer in the regulation market specifying the type of the Ancillary Service required as well as the desired power demand profile for that specific customer. If the customer accepts the signal, it is required to adjust its power demand after the profile received. In our vision the DCs are great candidates for providing Ancillary Services when requested because of their huge flexibility given by their positioning at the crossroads of energy, thermal and digital information networks, thus having the opportunity to transform themselves into key players within their local, sustainable energy systems. Thus we propose a DC optimization methodology that aims at integrating the DCs into the Smart Grid scenarios and enacting them to respond to various Ancillary Services Requests coming from Distribution System Operator (DSO). The methodology is leveraging on flexibility mechanisms such as load time shifting, alternative usage of non-electrical cooling devices (e.g. thermal storage) or charging/discharging the electrical storage devices, etc. As result of using our optimization methodology the DCs could: (i) shape their energy demand to provide additional load following reserve for large un-forecasted wind ramps, (ii) shed or shift energy demand over time to avoid an coincidental peak load in the grid, (iii) feed in the smart energy grid the energy (either power or heat) produced by turning on their backup fossil fuelled generators, despite quite inefficient and above all highly pollutant, gaining a net financial reward by the energy provider, which, in that way will avoid to turn on fossil-fuelled standby power generation plants. In this way energy providers can leverage on financial incentive as a concrete way to involve DCs in more systemic energy efficiency programs. We have conducted experiments via numerical simulations based on real world traces of DC operation from production systems which highlight the potential of our methodology to optimize DC energy consumption and to shape its profile to provide Ancillary Services. The rest of the paper is structured as follows: Section II shows related work, Section III presents the proposed optimization methodology leveraging on flexibility
mechanisms defined, Section IV presents numerical simulation based experiments and results, while Section V concludes the paper and presents the future work. 2 Related Work Recent studies have shown that DCs are great candidates for participating in demandresponse programs due to the following reasons [1]: (i) are large energy consumers or producers if they have on-site renewable energy production facilities, (ii) are highly automated and may respond fast to demand response (DR) signals and (iii) are characterized by a large energy flexibility due to the nature of their hardware components and workload executed. One of the most comprehensive studies describing the potential of different DC s hardware components and strategies providing demand response was released by Lawerence Berkeley National Laboratories [2]. The strategies are referring to: shutting down IT equipment, load shifting or queuing IT Jobs, temperature set point adjustment, load migration and IT equipment load reduction. Even though the results were promising the DCs being able to adjust the demand profile with 10-12% the approach fail to take into consideration the correlations and combination among strategies to obtain a stronger response as well as new technologies such as non-electrical cooling or batteries. Also even if the report shows that it is feasible for a DC to respond to DR signals no DC operation optimization algorithm is presented. We have classified literature techniques for optimizing DCs operation to participate in demand response programs and provide Ancillary Services according to the leveraging mechanism used in: (i) techniques based on load migration and consolidation, (ii) techniques exploiting electrical batteries and (iii) techniques based on cooling system intensity adjustments and heat removal strategies. Time and spatial load migration and consolidation techniques are usually used for voluntary reduction of DC s energy demand. Authors of [14] evaluate the DCs that offer Ancillary Services in form of voluntarily load reduction using an analytical profit maximization framework and propose an optimization technique based profit maximization strategy. The authors propose a mathematical model that includes: DC s internet service revenue, the cost of electricity and the compensation it may receive by offering Ancillary Services. Furthermore, it takes into consideration the servers power consumption, DCs Power Usage Effectiveness (PUE), workload statistics and SLAs. This approach interferes with the workload behaviour, resulting in performance degradation. Also, it does not consider any other form of energy flexibility mechanisms, such as, electrical storage devices, non-electrical cooling devices, etc. In [3] the authors present a solution for minimizing the electrical bill in a smart grid that employs both day ahead dynamic pricing and regulation signals. At the beginning of a billing period (several minutes to one day), the market participants (DCs) receive a regulation signal which specifies the trend of energy consumption during that period. During the billing period, DSO updates the initial trend, by sending regulation signals. A two tier DC controller is implemented, which performs resource allocation and schedules task dispatch, achieving optimality in minimizing the overall cost. A more
complex approach is presented in [1]. It takes advantage of two DC flexibility mechanisms: workload shifting and local generation (local diesel generators and local renewable energy). Using these mechanisms, algorithms are developed in order to avoid the coincidental peak and reduce the energy costs. They rely on the prediction of coincident peak occurrence based on historical data to optimize the workload allocation and local generation and to minimize the expected cost. The authors of [15] present a dynamic pricing system for a federation of DCs and use a distributed constraint optimization solver to negotiate a mutually optimal price. Workload relocation between DCs is used to meet the energy need at various DCs from different geographic locations. This technique is limited by several factors, such as DC capacity, workload security and SLAs, as well as extra energy needed for data transportation. Lately with the advent of batteries technologies and due to the fact that they don t pose any workload degradation overhead they are starting to be considered as important resources for helping DCs to participate in demand response. One of the first approaches of using batteries as a flexibility element within the DC is presented in [12]. The authors propose exploiting the batteries as an energy buffer with three functionalities: first it can shave power peaks; secondly it can store energy when it is cheap and third it can increase the DC power consumption when requested. In [13] a technique is proposed for balancing and keeping the DCs peak power under a given threshold (because of the electricity pricing) but in the same time allowing DCs to respond to the regulation control signals that may request an increase in power consumption. A detailed energy storage scheme is presented in [10] in which uninterrupted power supply (UPS) units are used as energy storage devices. Based on the existence of delay-tolerable workload, it tries to reduce the time average electricity costs using Lyapunov optimization. The control algorithm will decide at each time moment how much energy to draw from the grid, how much of it to save into the energy storage and how much energy to draw from the storage such that the time average cost of these operations is minimized. Even though efforts have been made to reduce the cooling system energy consumption it is still is responsible for as much as 50% of the energy consumption in a DC thus it is important resource for energy flexibility. In [6] a strategy of optimizing the energy consumption of the cooling system, by evaluating the time-varying power prices is presented. The system checks the prices in the hour-ahead market, and precools the air masses. In this way, later, when the power prices increase, the thermal masses can absorb heat for a given period of time. The electrical cooling system of a DC is detailed in [9], where a hybrid cooling system composed of traditional CRAC cooling, free air cooling and liquid cooling is presented. The authors propose a power optimization scheme that combines the different cooling techniques and dynamically adjust the cooling source to minimize the overall power consumption. Furthermore, the heuristic is extended to handle a network of DC by dynamically dispatching the incoming requests among a network of DC with various cooling systems. New approaches are consider the DCs energy producers and target the reuse of the heat generated by the DC for heating nearby residential or commercial buildings is presented in [16]. Under certain conditions, the extra energy from the DCs renewable sources may be fed back to the grid. In this context, the authors propose a solution that aims to
integrate the DCs with smart grids. To prove their results the authors use a grid simulation integrated with a DC operation simulation in which the DC observes the grid and adapts its internal state to meet grid s conditions. In this context our approach paves the way for next generation energy sustainable DC which is able to optimize its overall energy consumption considering the installed hardware components and available flexibility mechanisms in a holistic and integrated manner. To increase and exploit the potential of DC to provide Ancillary Services along with load time shifting load or using diesel generators to feed energy to the grid we will also investigate and define flexibility mechanisms non electrical cooling devices such as the thermal storage as well as electrical storage devices. 3 DC Optimization Methodology Ancillary Services address short-term imbalances in the grid and the DC as a demand response player has the role of providing help in balancing the grid on short time basis. We have defined an optimization methodology which exploits the available levels of electrical energy demand flexibility offered by the following DC s components: IT Computing Resources (i.e. servers), Electrical Cooling System, Electrical Storage System (i.e. batteries) and Thermal Storage System [21] (allow for excess thermal energy to be collected for later use). The overall goal is to provide an optimal capacity and operational planning for the DC to provide Ancillary Services if requested. It uses a discrete modelling of DC as a system and defines the current energy state of each component at timeslot t as a function of previous states at timeslots t-1, t-2 t-k. The flexibility mechanism for the IT Computing Resources energy demand is leveraging on the time-shifting of delay tolerant workload. The real-time workload must be executed as soon as it arrives, while the delay-tolerant workload can be executed in the future but no later than a given deadline. The DC s energy demand is reduced at timeslot with the amount of energy needed to execute the delay-tolerant load that is shifted at timeslot while the DC s energy demand at timeslot is increased with the amount of energy needed to execute the delay-tolerant load shifted from timeslot. To decide on the optimal timeslot and amount of delay tolerant workload to be shifted we define a model for estimating the energy consumption of the DC IT servers for a given workload configuration. We define a workload scheduling matrix ( ) where its element represents the percentage of the delay-tolerant workload, with representing the arrival timeslot, the execution timeslot and T the maximum dimension of the time window based on the Ancillary Service response length: ( ) (1) Due to the fact that the execution of a delay-tolerant workload cannot be scheduled before its arrival timeslot, the scheduling matrix is an upper triangular one having zero values under its main diagonal. A matrix row represents all the workload re-
ceived at timeslot and divided in percentages associated to the delayed execution timeslot until its deadline. As a consequence, the sum on each matrix row is 1. A matrix column represents all the workload scheduled for execution at timeslot. By summing all the workload elements of a column and translating them in energy values we obtain the estimated amount of energy needed to execute the delay-tolerant workload at timeslot (relation 2), while the energy demand for executing all DC workload is calculated using relation 3: ( ) ( ) (2) ( ) ( ) ( ) (3) The flexibility mechanism of electrical cooling system is leveraging on the usage of non-electrical cooling systems such as the Thermal Aware Storage (TES). The DC s energy demand at timeslot is decreased with the amount of energy discharged from the TES (as consequence of lowering the intensity and demand of the electrical cooling system), while at timeslot the DC s energy demand is increased with the amount of energy charged in TES (as consequence of increasing the intensity of the electrical cooling system to overcool the TES tanks). To estimate the amount of flexibility that can be offered by the TES we compute the TES level at timeslot, denoted by ( ) based on the previous state and the actions of charging and discharging at timeslot, denoted by ( ) and ( ), respectively (see relation 4). We have considered the following characteristics of the TES device: charge loss factor during operation ( ), discharge loss factor during operation ( ), and time discharge factor when it is not operated ( ). ( ) ( ( ) ( ) ( ) ( ) ( )) (4) To estimate the level to which the electrical cooling system can be lowered due to the fact that its operation is compensated by TES discharging we have estimated how much power is used by the electrical cooling system. We use [7] assumption that all the electrical power consumed by the DC IT resources to execute the workload is transformed into heat and must be dissipated by the cooling system. To estimate the actual cooling power needed to deal with the dissipated heat, the formula provided by [6] is used and enhanced with the TES flexibility mechanism described above where COP is the cooling system Coefficient of Performance: ( ) ( ( ) ( )) (5) The flexibility mechanism defined for an energy electrical storage device (ESD) is based on reducing at timeslot the DC energy demand with energy discharged from the batteries and increasing at timeslot + the DC energy demand with the energy charged in batteries. A DC is equipped with batteries that store energy to cover its energy consumption needs for a short period of time. The state of the art ones have a higher charge-discharge life-cycle that allows them to be used more fre-
quently thus offering a certain level of flexibility for the DC energy demand. Being similar with the TES, we need also to estimate the amount of energy that is discharged from the devices when they are used to power the DC ( ( )) and the amount of energy that is charged in the devices from the DC surplus energy ( ( )) this time taking into consideration the operating parameters of the ESD, such as: energy losses incurred during both charge and discharge cycles and also during the time the device is not used (we denote the charge loss, discharge loss and time discharge factors by, and, respectively), and the percentage of maximum energy removal during a discharge, Depth of Discharge ( ). ( ) ( ( ) ( ) ( ) ( ) ( )) (6) Using the above presented model at each timeslot t we can determine the DC energy demand as: ( ) ( ) ( ) ( ) ( ) ( ) (7) To shift the energy demand profile for providing Ancillary Services the following optimization function need to be minimized: ( ( ( ) ( )) ( ( ) ( )) ( ) ( ( ) ( )) ( ) ) (8) The optimization function aims at shedding the DC energy demand profile to match a requested energy profile by the DSO for the response interval while reducing the energy consumption alteration impact on the rest of the optimization time window: before and after DC s response interval. Thus the function computes the normalized distance between the DC energy demand curve ( ), and the reference values requested by the DSO ( ) during the demand response period on one side and the DC energy demand baseline (i.e. expected energy consumption estimated based on the workload the DC has to execute) ( ) for the rest of the prediction window on the other side. Furthermore, in order to minimize the DC energy consumption changes from the initial baseline, the distances outside the demand response interval are weighted with a function that increases with the distance from the interval, thus assuring that the changes in the DC energy consumption from the initial baseline are kept as close as possible to the demand response period. 4 Use Case Evaluation To demonstrate the viability of our proposed optimization methodology, numerical simulation based experiments have been carried out, with a view to subsequently apply and validate the approach in the coming months in real pilot DCs of GEYSER
EU FP7 project [19]. To that end, we have developed a simulation environment in which the hardware systems characteristics and operation (see Table 1) of a real DC are modelled. The workload energy demand was taken from the IT power consumption logs of the DC [18] considering 5 minutes samples normalized using the maximum power consumption of modelled DC. Component Electrical Cooling System IT Computing Resources Electrical Storage System Thermal System Storage Table 1. DC s hardware components characteristics Hardware Characteristics Simulated Cooling Capacity = 4000 kwh, Minimum Cooling Load = 200 kwh, Maximum Cooling Load = 2000 kwh, COP Coefficient = 3.5 11000 SERVERS with: P_MAX = 325 W, Memory, Processor, Hard Drive = RAM 8 GB, CPU 2.4 GHz, HDD 1Tb Charge Loss Rate = 1.2, Discharge Loss Rate = 0.8, Energy Loss Rate = 0.995, Max Charge & Discharge Rate = 1000 kwh, Max Capacity = 1000 kwh Charge Loss Rate = 1.1, Discharge Loss Rate = 0.99, Energy Loss Rate = 0.999, Max Charge & Discharge Rate = 1000 kwh, Maximum Capacity = 3000 kwh The proposed methodology was used to shape the energy demand profile of the simulated DC aiming at providing the Ancillary Services described in Table 2. The services are distinguished by their following requirements: (i) how fast the DC must respond if it can fulfil the requested service (response time), (ii) the minimum length of the DC response, and (iii) the hardware resources on which the DC is leveraging on generating the requested output considering factors such as the modelled hardware devices inertia in providing flexibility. Service Name Regulation Scheduling Reserve Description Table 2. Ancillary Services the DC may provide Increase demand for large un-forecasted renewable energy production Shed or shift energy consumption over time to match a requested profile Provide fast ramping power Time to full Response Response Length DC mechanism 20 min 1 hour Load time shifting, ESDs, electrical cooling system adjustments using TES 10 min > 1 hour Load time shifting, ESDs, electrical cooling system adjustments using TES <10 min 30 min ESDs and diesel generators The optimization objective function (see relation 8) needs to be minimized considering the following input data for the response length time window: requested energy profile for the prediction window associated with the Ancillary Service signal, DC energy demand baseline without optimization and DC predicted energy demand. To solve the minimization problem Mixed Integer Nonlinear Programing [20] was used.
4.1 DC Providing Regulation Ancillary Service In this scenario we assume that in the energy grid there is large un-forecasted renewable energy production that may threaten the grid stability. In consequence the DSO sends a regulation signal to the DC connected to the grid at 12pm requesting to increase its energy demand between 12:25pm and 1:15pm and consume as must as possible the energy surplus. Figure 1 top presents the DC energy demand baseline with orange line and requested profile with blue line. In response DC takes advantage of our methodology to optimize its demand using the available flexibility mechanisms such that it consumes more energy in the requested period (see Figure 1 bottom). Fig. 1. DC Providing Regulation Ancillary Service By applying the optimization techniques proposed, the DC energy demand profile is adapted to the Regulation Ancillary Service signal request. Consequently, it can be seen that most of the delay tolerable workload arrived in the interval 12-12:25pm has been delayed to fill the energy demand gap in the interval 12:25-1:15pm. Also, the electrical cooling system is used more in this interval to overcool the TES, such that they can be used later to reduce the energy consumption if needed by substituting the electrical cooling system. As a result, the DC energy demand was increased by 34% from original baseline on the service response length.
4.2 DC Providing Scheduling Ancillary Service In this scenario we assume that in the Smart Grid there is predicted energy consumption coincident peak and in order to avoid it the DC is requested to shed or shift energy consumption over time. In consequence the DSO sends a scheduling signal to the DC connected to the grid at 12pm requesting to decrease its energy demand between 12:30pm and 2pm. Fig. 2. DC Providing Scheduling Ancillary Service Figure 2 presents the DC energy demand baseline; the requested profile with blue line and the DC optimized energy demand profile obtained using our methodology and the flexibility actions used. As it can be noticed using our methodology the DC is able to reduce the energy consumption over the service response period with 32% to meet the request. Most of the delay tolerant workload from the service response period is delayed after 2:30pm. The electrical cooling device is used less with 20% during the response period by leveraging on the TES device which was prior overcooled.
Also no charging batteries actions are taken during this period. Total amount of flexible energy shifted to provide the requested services is 8.43 MWh. 4.3 DC Providing Reserve Ancillary Service In this scenario we assume that the DSO in order to maintain the (local) reactive power balance under normal conditions and to ensure a stable grid operation with adequate voltage control requests the DC to provide reserve Ancillary Service. In consequence it sends a reserve signal to DC to produce energy using its diesel generator (the greener biomass-fuelled generators are not yet considered for DC backup power generation) and feed it back to grid. To increase the amount of energy fed to the grid the DC uses our optimization methodology to shift as much as possible the energy demand away from the time interval when the service is expected by using at minimum the electrical cooling system and shifting the execution of delay tolerant workload. Also it leverages on the batteries to feed extra power to the grid. Fig. 3. DC Providing Scheduling Ancillary Service Figure 3 presents the optimized DC energy demand profile while Figure 4 presents the diesel energy generated and feed to the grid. We have considered that the Reserve Ancillary Service signal is received at 12:00pm, the DC will provide fast ramping
power starting from 12:10pm and the service response length is 30 minutes until 12:40pm. During the reserve service duration the using our methodology the DC manages to shift about 4.15 MWh of flexible energy resulting in a temporary decrees of energy consumption with 15%. Also the amount of energy feed to the grid is 8.79 MWh out of which 52% is represented by the energy produced using the diesel generator, while the rest of 4.15 MWh is energy saved as result of demand profile optimization. Fig. 4. DC Energy Feed to the Grid 5 Conclusion This paper presented an energy demand optimization methodology that aims at enacting DCs connected to the smart grid to provide Ancillary Services on demand. The methodology is leveraging on flexibility mechanisms defined for DC hardware components such as load time shifting, alternative usage of non-electrical cooling devices (e.g. thermal storage) or charging/discharging the electrical storage devices, etc. Simulation results validate the DC potential for shaping its energy profile to meet goals of diverse energy networks and provide three types of Ancillary Services: Regulation, Scheduling and Reserve. The approach is about to be validated within the context of four operational DCs of the GEYSER project (Pont Saint Martin and Terni in Italy, Alticom in the Netherlands and RWTH Aachen in Germany). 6 Acknowledgment This work has been conducted within the GEYSER project Grant number 609211 [19], co-funded by the European Commission as part of the 7th Research Framework Programme (FP7-SMARTCITIES-2013).
7 References 1. Z. Liu, A. Wierman, Y. Chen, B. Razon, N. Chen, Data Center Demand Response: Avoiding the Coincident Peak via Workload Shifting and Local Generation. In: ACM SIGMETRICS/international conference on Measurement and modeling of computer systems, pp 341-342, ACM, New York (2013) 2. G. Ghatikar, V. Ganti, N. Matson, M. A. Piette: Demand Response Opportunities and Enabling Technologies for Data Centers: Findings from Field Studies, 2012, http://eetd.lbl.gov/sites/all/files/lbnl-5763e.pdf 3. M. G.-Gol, Y. Wang, M. Pedram, M. Hsieh: An Optimization Framework for Data Centers to Minimize Electric Bill under Day-Ahead Dynamic Energy Prices While Providing Regulation Services. In: Green Computing Conference (IGCC), pp 1-9, IEEE, Dallas (2014) 4. J. Koomey. Growth in data center electricity use 2005 to 2010. Oakland, CA: Analytics Press. August, 1, 2010. 5. O. Ma, N. Alkadi: Demand Response for Ancillary Services. In Smart Grid, IEEE Transactions on (Volume:PP, Issue: 99 ), IEEE, 2013 6. Y. Zhang, Y. Wang, and X. Wang: TEStore: Exploiting Thermal and Energy Storage to Cut the Electricity Bill for Datacenter Cooling. In: 8th international conference and workshop on systems virtualization management, pp 19-27, Las Vegas (2012). 7. Q. Tang: Sensor-Based Fast Thermal Evaluation Model for Energy Efficient High- Performance Datacenters. Intel Corporation, In: Intelligent Sensing and Information Processing (ICISIP) 2006, pp 203-208, IEEE, Bangalore (2006). 8. R. L. Sawyer: Calculating Total Power Requirements for Data Centers, Whitepaper, Online at http://www.apcmedia.com/salestools/vavr-5tdtef/vavr- 5tdtef_r1_en.pdf?sdirect=true 9. L. Li, W. Zheng, X. Wang: Coordinating Liquid and Free Air Cooling with Workload Allocation for Data Center Power Minimization. In:11th International Conference on Autonomic Computing (ICAC 14), USENIX, Philadelphia (2014). 10. R. Urgaonkar, B. Urgaonkar, M. Neely, and A. Sivasubramaniam: Optimal power cost management using stored energy in data centers. In ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems, pp 221-232, ACM, New York (2011) 11. W. Zheng, K. Ma, and X. Wang: Exploiting Thermal Energy Storage to Reduce Data Center Capital and Operating Expenses. In: 20th International Symposium on High Performance Computer Architecture (HPCA), IEEE, Orlando (2014) 12. S. Govindan, A. Sivasubramaniam, and B. Urgaonkar: Benefits and limitations of tapping into stored energy for datacenters. In: Proceedings of the 38th annual international symposium on Computer architecture ( ISCA), pp 341-352, IEEE, San Jose (2011) 13. B. Aksanli and T. Rosing: Providing regulation services and managing data center peak power budgets. In: Design, Automation and Test in Europe Conference and Exhibition (DATE), pp 1-4, IEEE, Dresden (2014)
14. M. Ghamkhari and H. Mohsenian-Rad: Data Centers to Offer Ancillary Services. In: Proceedings of IEEE international conference on smart grid communications (SmartGridComm), IEEE, Tainan City (2012) 15. Y. Li, D. Chiu, C. Liu, L. T.X. Phan, T. Gill, S. Aggarwal, Z. Zhang, B. Thau Loo, D. Maier and B. McManus: Towards Dynamic Pricing-Based Collaborative Optimizations for Green Data Centers. In: Data Engineering Workshops (ICDEW), pp 272-278, IEEE, Brisbane (2013) 16. S. Janacek, G. Schomaker, W. Nebel: Data Center Smart Grid Integration Considering Renewable Energies and Waste Heat Usage: Energy-Efficient Data Centers,LNCS 8343, 2014, pp 99-109 17. Y. Chen, et. al.: The Case for Evaluating MapReduce Performance Using Workload Suites. In: Modeling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS), pp 390-399, IEEE, Singapore (2011) 18. Nord Pool Spot Market Data, http://www.nordpoolspot.com/marketdata1/elspot/area-prices/all1/hourly/?view=table 19. GEYSER Green Energy Data Centers as Energy Prosumers in Smart City, http://www.geyser-project.eu/ 20. LINDO Software for Integer Programming, Linear Programming, Nonlinear Programming, Stochastic Programming, Global Optimization, http://www.lindo.com/ 21. W. Zheng, K. Ma, and X. Wang, Exploiting Thermal Energy Storage to Reduce Data Center Capital and Operating Expenses, http://www2.ece.ohiostate.edu/~xwang/papers/hpca14.pdf