Online Control of Datacenter Power Supply under Uncertain Demand and Renewable Energy

Transcription

1 Online Control of Datacenter Power Supply under Uncertain Demand and Renewable Energy Wei Deng, Fangming Liu, Hai Jin, Xiaofei Liao Services Computing Technology and System Lab, Cluster and Grid Computing Lab School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, , China {wdeng, fmliu, hjin, Abstract Modern Cloud Service Providers (CSPs) equip their Datacenter Power Supply System (DPSS) with multisources to mitigate power cost, carbon emission, and power outage: (1) multi-markets grid with time-varying energy prices, (2) finite capacity of Uninterrupted Power Supply (UPS), and (3) certain volumes of intermittent renewable energy. With the presence of uncertain renewable sources and datacenter power demand, CSPs have a critical challenge: how to design systematical online control policies that best utilize different characteristics of multisources in a complementary manner to deliver reliable energy to datacenters while minimizing DPSS operation cost. Based on a stochastic optimization model that captures characteristics of DPSS, we apply two-stage Lyapunov optimization to design and analyze an online DPSS control algorithm (OnDPSS). OnDPSS makes decisions on fully utilizing renewable energy, two-timescales power purchasing, and UPS charging/discharging without requiring substantial statistics of system dynamics. Our mathematical analyses and one-month trace-driven simulations have demonstrated both the optimality (in terms of tradeoff between minimization of DPSS operational cost and constraint satisfaction on datacenter availability and UPS lifetime) and system stability (in terms of robustness to time-varying power demand and supply) achieved by OnDPSS algorithm. I. INTRODUCTION AND RELATED WORK The fast proliferation of the Internet-scale systems and cloud computing services has promoted the massive, geographically distributed datacenters. Large cloud service providers (CSPs) have three major problems during the operation of such datacenters: (1) skyrocketing power consumption and annual electricity bills, such as Google (> 1, 120GW h and > $67M) and Microsoft (> 600GW h and > $36M) [1]. (2) Serious environmental impact. Reportedly, IT carbon footprints occupy 2% of global CO 2 emissions [2]. (3) Unplanned power outages. For instance, Amazon experienced two outages in June 2012 in its US-East-1 region, which were triggered by a series of failures in the power infrastructure [3]. To mitigate such power cost, carbon emission and power outage, modern CSPs usually equip their datacenter power supply systems (DPSS) with multisources in a complementary manner, as illustrated in Fig. 1. First, the grid is still the dominant way to power datacenters for its stable and abundant supply. CSPs can procure power from multiple timescales electricity markets with different energy prices, ranging from long-term market to real-time market [4] [8]. Second, Uninterrupted Power Supply (UPS) is used as the most common way to back up DPSS when there s a grid outage [9]. Recent studies have shown additional benefits of UPSs for reducing power The Corresponding Author is Fangming Liu (fmliu@hust.edu.cn). Uncertain Demand d(t) Long-term ahead planning Datacenter Power Supply System Online Control DPSS to Match Demand Real-time Balancing Supply g(t) ylt(t) yrt(t) Discharge D(β(t)) Long-term Real-time UPS Renewables Multi-peroid Grid Markets Battery Charge R(t) Automatic Transfer Switch(ATS) CSP Datacenters Fig. 1. System model of matching uncertain power demand and supply for CSP to minimize operational cost. cost, by shaving peaks of power demand or shifting demand away from high tariff periods [9], [10]. Third, many CSPs have started to green their datacenters with renewable energy, such as solar and wind, for lower power cost and carbon footprints [2]. In practice, CSPs can automatically switch their power sources via Automatic Transfer Switch (ATS) [11]. When operating such DPSS, there are several key control decisions to be made: (1) How much power to be purchased from the long-term grid market and real-time market, respectively? (2) How to fully leverage the available renewable energy? (3) How to opportunistically use the UPS battery to reduce power cost? With the presence of uncertain power supply and demand, it is challenging to take different characteristics of multiple sources into the consideration of the above decision-making for delivering reliable energy. On the demand side, due to time-varying resource usages of diverse application workloads, a datacenter s power demand is usually variable and uncertain. On the DPSS side, due to the variable energy prices, purchasing power from the grid with multiple markets results in time-varying costs. Furthermore, the intermittent and unpredictable nature of renewable energy aggravates supply-side uncertainty. In addition, the charging/discharging of finite capacity of UPS impacts UPS lifetime and datacenter availability in terms of the continued operation of datacenter [9], [10]. Although many existing works attempted to handle the supply or demand uncertainties in smart grid and datacenters, they either (1) focused on managing the energy supply by assuming a priori knowledge of energy demand [5], [7], [8], [12], or (2) required substantial statistics of the system dynamics with excessive computational complexity [4], [6], or (3) limited to only a single-day optimization of the system [6], or (4) relied on complex forecast techniques of renewable energy [13], [14]. On the contrary, we seek to design online DPSS control policies for a long-term running under uncertain power demand and supply, without requiring substantial

2 system statistics and forecast techniques. Based on a stochastic optimization model that captures time-varying power demand, renewable energy supply, finite UPS and two-timescale grid markets, we derive an online DPSS control algorithm (OnDPSS) by applying two-stage Lyapunov optimization techniques [15], [16]. OnDPSS makes online decisions on fully utilizing renewable energy, twotimescales power purchasing and UPS charging/discharging in a complementary manner to minimize DPSS operation cost. OnDPSS can approach the optimal solution within provable O(1/V ) deviation without requiring substantial statistics of system dynamics. In particular, the control parameter V serves as a control knob that allows CSPs to adjust V that controls the trade-off between minimization of DPSS operation cost and violation of constraints on datacenter availability, UPS lifetime and power capacity. Our mathematical analyses and one-month trace-driven simulations have demonstrated both the optimality (in terms of a well balanced trade-off between DPSS operation cost and constraints violation) and system stability (in terms of robustness and adaptivity to time-varying power demand and supply) achieved by OnDPSS. II. DPSS: OPERATIONAL MODEL AND OBJECTIVE The DPSS system operates in a discrete-time model. Without loss of generality, time is divided into K(K N + ) coarsegrained slots of length T in accordance with the length of longterm ahead market, e.g., days or hours [6]. Each coarse-grained time slot is further divided into N T (N T N + ) fine-grained time slots of length T 1. Empirically, each fine-grained time slot T 1 could be 15 or 60 minutes per which the datacenter can adjust its power control strategies [7], [16]. A. Online Control Decisions As illustrated in Fig. 1, the operation of DPSS includes three key control decisions: 1) Long-term ahead power procurement: At each coarsegrained time slot t = kt (k = 1, 2,..., K), the CSP observes the amount of available renewable energy, denoted as g lt (t), and power demand of a datacenter, denoted as d lt (t). Then, the CSP decides how much amount of power y lt (t) to be purchased from the long-term ahead market at a price p lt (t) for each period τ [t, t + T 1]. 2) Real-time power procurement: At each fined-grained time slot τ [t, t + T 1], the actual power demand d(τ) and available renewable energy g(τ) can be readily observed by the CSP. If the long-term ahead purchasing and the renewable energy are enough to meet the current power demand, i.e., y lt (t) + g(τ) d(τ), then no further procurement is needed. Otherwise, the CSP decides how much additional power y rt (τ) to be purchased from real-time market with a price p rt (τ). 3) Real-time UPS charging/discharging: We assume that CSP can switch between different power sources via automatic transfer switch automatically [11]. If no real-time procurement is needed, then R(τ) = y lt (t) + g(τ) d(τ) will be the excessive energy that can be used to charge the UPS, where R(τ) represents the charged battery energy. Otherwise, the CSP has to make a decision of whether to opportunistically discharge energy D(β(τ)) from the battery along with realtime power procurement to fulfill the current demand: y lt (t) + y rt (τ) + D(β(τ)) + g(τ) = d(τ), (1) where D(β(τ)) denotes the amount of UPS energy discharged at the depth-of-discharge (DoD) level of β(τ) (β(τ) [0, 1]). Let m(t) denote the UPS energy level. We assume that the efficiencies of UPS charging and discharging are the same, denoted by η [0, 1], e.g., η = 0.8 means that only 80% of the charged or discharged energy is useful when charging or discharging. The dynamics of UPS level m(t) can be expressed as below [10], [12]: m(t + 1) = m(t) + ηr(t) D(β(t))/η. (2) B. Online Control Challenges and Constraints There are a series of challenges in decision-making: 1) Power procurement accuracy and cost: Intuitively, the closer to real-time, the CSP has more accurate information of available renewable energy and datacenter power demand, and hence can make more accurate decision on power purchasing [6], [8]. However, it is known that the real-time energy price is more expensive (e.g., p rt (t 1 ) > p lt (t 2 ) if t 1 < t 2 ) [5], [6], [8]. In practice, the price of electricity in real-time markets tends to be higher on average than that in long-term ahead market [5]. The rationale is that upfront payment is associated with cheaper contract prices in the long-term markets. Hence, when procuring power in two-timescale markets (long-term and real time markets), the CSP should make the best tradeoff between procurement accuracy and power cost. 2) Grid capacity limit: We assume that the maximum amount of power that the CSP can draw from the grid in any time period is limited by P grid : 0 y lt (t) + y rt (t) P grid. (3) 3) Datacenter availability constraint: It is extremely critical to guarantee desired datacenter availability with finite capacity of UPS. For instance, ebuff [9] always leaves fiveminutes reserved energy in batteries to ensure datacenter availability. We assume that UPS has a capacity of M UP S, and we choose to store M min minutes of energy to ensure datacenter availability: M min m(t) M UP S. (4) 4) UPS recharge/discharge constraint: In practice, battery has constraints on the maximum amounts of power by which we can recharge or discharge the battery [10]: 0 D(β(t)) D max, 0 R(t) R max. (5) 5) UPS operational cost and lifetime constraint: It has been practically shown that UPS lifetime is a decreasing function of DoD and charge/discharge cycles [17]. The cost of repeated operation of the battery is a function of how often/much it is charged and discharged. Each recharge and discharge operation incurs a cost of C r and C d (β) (at DoD level of β) in dollars, respectively. If a new UPS costs C ups dollars to purchase and it can sustain L ups charge/discharge cycles at maximum DoD, then C r = C ups /L ups, C d (β) =

3 βc ups /L ups. During the long runtime horizon t KT, the maximum allowable discharging number N max satisfies: a(τ) N max, (6) where a(t) = 1 if D(t) > 0, otherwise a(t) = 0. We simplify each charging/discharging cost as a(t)c r (ignoring the impact of DoD β on discharging cost). C. Stochastic Cost Minimization Problem Since the primary costs for renewable energy generation are construction costs such as deploying solar panels and wind turbines, their operational cost is negligible [11]. At time t, the CSP s cost is the power procurement cost plus UPS charging/discharging cost, i.e., Cost(t) y lt (t)p lt (t) + y rt (τ)p rt (τ)) + a(τ)c r. We seek to design an online DPSS control algorithm for solving the following stochastic cost minimization problem: 1 Cost av lim inf E[Cost(τ)](7) s.t. t : constraints (1)(2)(3)(4)(5)(6). min y lt,y rt,d(β),r Lemma 1: In every optimal solution of the optimization problem (7), it holds that y rt (τ) 0, or p rt (t) 0 for τ. Corollary 1: There exits an optimal solution of the above optimization problem (7) wherein y rt (τ) 0 for τ. All complete proofs of Lemma 1, Corollary 1 and Theorems (Sec. III) are referred to our detailed technical report [18]. Remark: The above implies that real-time purchasing is unnecessary (y rt (τ) = 0) in the optimal condition, where all the information about datacenter workloads, renewable energy and energy prices are known in advance. However, this would be too idealized. Without such statistics in practice, how to perform real-time purchasing and battery operation becomes an important problem that we should investigate. To design a flexible and robust online control policy, traditional dynamic programming [6] or Markov decision process [4] require substantial statistics of the system dynamics with excessive computational complexity. In contrast, the recently developed Lyapunov framework is shown to enable the design of online control algorithms for such constrained optimization of time-varying systems without requiring a priori knowledge of the workload and cost statistics [15]. In particular, our above model of the two-timescale power purchase and delivery structure well fits the two-timescale Lyapunov optimization [16], that can enable us to perform two levels of control strategies for two levels of granularity. Therefore, we choose to design our online control algorithm based on two-timescale Lyapunov optimization. III. ONDPSS: ALGORITHM DESIGN AND ANALYSIS A. Lyapunov Optimization Framework To solve the problem (7), we first define a set of virtual queues for the constraints that are captured by our above model, so as to transform constraints satisfaction into queue stability problem. The rationale is that in Lyapunov optimization the constraints are satisfied when their corresponding virtual queues are stable [15]. To ensure DPSS constraints of datacenter availability (4), UPS discharge rate constraint (5), and UPS lifetime (6), we define virtual queues with their update equations as follows: H(t + 1) = max[h(t) m(t), 0] + M min, (8) J(t + 1) = max[j(t) N max, 0] + t a(τ), (9) Z(t + 1) = max[z(t) D max, 0] + D(β(t)). (10) We can note that: H(t + 1) H(t) m(t) + M min. Taking expectations on both sides and summing it up from t = 0 to KT 1, we obtain: E[H(t)] E[h(0)] KT 1 t=0 E[H(t)] + T M min. Dividing both sides by t, and taking t, we can see that H(t)/t 0, and thus the constraint M min m(t) follows. Similarly, we verify that the constraints of discharge rate (5) and UPS lifetime (6) will be ensured as well. Let Q(t) = [m(t), H(t), J(t), Z(t)] be a concatenated vector of the actual and virtual queues. We define a quadratic Lyapunov function as: L(Q(t)) 1 2 [m2 (t) + H 2 (t) + J 2 (t) + Z 2 (t)]. Then, the T -slot conditional Lyapunov drift is defined as: (Q(t)) E[L(Q(t+T )) L(Q(t)) Q(t)]. The drift uses the queueing dynamics to capture how the control policies at each period affect the satisfaction of constraints (2), (4), (5) and (6). Stable drift will ensure the constraints. Following the Lyapunov framework of drift-plus-penalty algorithm [15], our control algorithm is designed to make decisions on y lt (t), y rt (t), D(β(t)) and R(t) to minimize an upper bound on the following drift-plus-penalty term every T slots: (Q(t)) + V E{ Cost(τ) Q(t)}, where the control parameter V is chosen according to the CSP to tune the tradeoff between DPSS cost minimization and constraint satisfaction. A weaker constraint satisfaction can reduce cost, yet can incur adverse effects on datacenter availability. With the presence of dynamic system demand and supply, we approximate future statistics as the current statistics [13], so as to avoid the complexity and substantial statistics requirement of forecasts, i.e., m(τ) = m(t), ĝ lt (τ) = g(t), d lt (τ) = d(t) and p rt (τ) = p rt (t) for t < τ t + T 1. Then, the following Theorem 1 gives the analytical bound on the decisions of power purchasing and UPS charging/discharging. Theorem 1: (Drift-plus-Penalty Bound) Considering the Lyapunov function L(Q(t)), to ensure the constraints in (1) (6), the drift-plus-penalty expression satisfies: T (Q(t)) + V E{ Cost(τ) Q(t)} (11) BT + +V E{ Cost(τ) Q(t)} m(τ)[r(τ)η D(β(τ))/η] Q(t)} H(τ)[M min m(τ)] Q(t)} J(τ)[ a(τ) N max] Q(t)} Z(τ)[D(β(τ)) D max Q(t)]}, where B = R 2 maxη 2 + D2 max η 2 + M 2 ups M 2 min 2 + N 2 max + D 2 max.

4 Remark: To minimize the right-hand-side (RHS) of (11), we design our online DPSS control algorithm (OnDPSS) as Algorithm 1. Specifically, OnDPSS is computationally efficient: each time it only has to solve a linear program with four variables (y lt (t), y rt (t), D(β(t)), R(t)) and six linear constraints in Eq. (1)(2)(4)(5)(6) without a priori statistical knowledge about system dynamics. We do not explicitly take the constraint (3) into consideration, as it is easy to automatically ensure grid capacity limit (3) and battery charge rate R(t) [0, R max ]. OnDPSS intends to make full utilization of renewable energy and battery to minimize power procurement. B. Performance Bound and Robustness Analysis In this section, we analyze the performance of OnDPSS algorithm in terms of the gap between the result achieved by OnDPSS and the theoretical optimal solution φ opt of cost minimization problem (7). Theorem 2: (Performance Bound): The average cost Cost av achieved by OnDPSS algorithm satisfies the following bound with any fixed control parameter V (V > 0) and longterm time slot T : 1 OnDP SS Costav lim inf E[Cost(τ)] φopt + B V, where B is given by Theorem 1. Remark: OnDPSS can approach the optimal solution of problem (7) within a deviation of B/V. CSPs can achieve an elastic cost by adjusting the control knob V, according to a desired tradeoff between DPSS cost minimization and constraints satisfaction. The time slot T decides how frequently OnDPSS performs actions of power procurement and battery charging/discharging. Interestingly, the dynamic UPS energy level reflects variable power demand and uncertain renewable energy supply. Since OnDPSS algorithm approximates future battery energy level as its current level, is it robust for OnDPSS algorithm to make its decisions based on the approximated UPS battery level m(t) that is different from the actual UPS capacity, due to uncertainties of datacenter power demand and supply? The following Theorem 3 demonstrates the robustness of OnDPSS. Theorem 3: (OnDPSS Robustness): We assume that the approximated UPS battery level m(t) and its actual level m(t) satisfy m(t) m(t) θ. Then, under OnDPSS algorithm: OnDP SS 1 Costav lim inf E[Cost(τ)] φopt + B ε V, where B is given by Theorem 1, and B ε = B + T θ(d max + R max + M ups + M min ). Here, D max and R max are the maximum amounts of UPS energy recharging and discharging, respectively; M ups and M min are the maximum and minimum UPS energy levels, respectively. Remark: By comparing Theorem 2 and Theorem 3, we observe that V varies positively along with the uncertainties (θ) of datacenter power demand and renewable energy. However, the satisfaction of constraints varies inversely versus these uncertainties. This implies that the robustness of OnDPSS is achieved at the expense of a tradeoff between operational cost minimization and constraints satisfaction. Algorithm 1: Online Control Algorithm for DPSS Operational Cost Minimization (OnDPSS). 0.8) Long-term ahead planing: At each time t = kt (k Z + ), observing available renewable energy g(t), datacenter power demand d(t), and UPS energy level m(t), CSP decides the optimal power procurement in the long-term market y lt (t) to minimize the expected cost as below: min V y lt (t)p lt (t) s.t. y lt (t) + g(t) d(t) + m(t), m(t) M UP S, 0 y lt (t) P grid. 2) Real-time balancing: At real-time τ [t, t + T 1], CSP performs real-time market procurement y rt(τ), and UPS discharging D(β(τ)) and charging R(τ), to minimize the RHS of (11): min V [y rt(τ)p rt(τ)) + a(τ)cr] + m(t)[r(τ) D(β(τ))] H(t)m(t) + J(t) a(τ) + Z(t)D(β(τ)) s.t. (4)(5)(6), y lt (t) + y rt(τ) + g(τ) + D(β(τ)) = d(τ), if y lt (t) + g(τ) > d(τ), R(τ) = y lt (t) + g(τ) d(τ). 3) Queue Update: Update queues using Eq. (2) (8) (9) (10). IV. ONDPSS: PERFORMANCE EVALUATION A. Real-World Traces and Experimental Setup First, to simulate the intermittent availability of renewable energy, we use solar energy data from the Measurement and Instrumentation Data Center (MIDC) [19]. Specifically, we use the meteorological data from Jan. 1st, 2012 to Jan. 31th, 2012 from central U.S.. Second, to simulate the variable electricity prices, we use the price traces from New York Independent System Operator (NYISO) [20]. Third, we use citizens electricity load data from NYISO to simulate the power demand of the CSP. Both traces are also from Jan. 1st, 2012 to Jan. 31th, According to the recent empirical experiments, we assume that the limits of UPS charging/discharging rates are D max = R max = 0.5MW, and charging/discharging costs are C r = C d = 0.1 dollars [10]. The minimum battery level M min is 5 minutes energy of UPS [9]. The maximum number of UPS charge/discharge cycles is L UP S = 5, 000 with 4-year lifetime constraint [9]. The efficiency of UPS charging/discharging is η = 0.8 [12]. We set the grid limit as P grid = 2MW [10]. B. Experimental Results We conduct sensitivity analysis on critical factors to characterize their impact on DPSS operational cost. 1) Impact of Control Parameter V : As shown in Fig. 2, to simulate the day-ahead power market, we fix T to be 24 time slots (one day). We conduct experiments with different values of V, which show that when V increases from 0.5 to 100, OnDPSS achieves a time-average cost that is closer to the optimal solution. This quantitatively confirms Theorem 2 that OnDPSS can approach the optimal solution within a diminishing gap of O(1/V ). This enables CSP to obtain an elastic performance bound by adjusting V within a desired weighted sum of cost and Lyapunov drift of constraints.

5 Day-Average Cost ($) OnDPSS Optimal V (T=24) Fig. 2. Impact of parameter V. Day-Average Cost ($) OnDPSS 1000 Optimal T (V=10) Fig. 3. Impact of time slot T. 2) Impact of Long-term Time Slot T : In Fig. 3, we fix V to be 10 and vary T from 3 time slots (3 hours) to 144 time slots (6 days), which is a sufficient range for exploring the characteristics of different timescales of long-term market. We observe that changing T has relatively less impact on the cost of operating DPSS. The fluctuation of the time-average cost is more notable when T becomes longer. This is because that the term B ε in Theorem 3 is proportional to T, which means the uncertainties of power demand and renewable energy increase with the increasing of T. As uncertainties can be alleviated by battery and two timescales power procurement, the timeaverage cost only fluctuates within [ 1.65%, +2.23%]. This corroborates Theorem 3 that, even with infrequent actions of DPSS operation, OnDPSS can save significant cost. 3) Impact of Battery Capacity: In Fig. 4, we compare the total average cost under different battery sizes (M UP S {0, 0.25, 0.5, 1}MW h) under two-timescale markets over the 31-days period with V = 10 and T = 24. It shows that UPS can bring in significant benefit, i.e., the total average cost decreases with the increasing of battery size. The rationale is that the UPS can offer additional free power charged from superfluous renewable energy to serve the loads, resulting in less purchasing costs. This verifies that the optimal solution is limited mainly by the battery capacity. 4) Impact of Grid Markets: In Fig. 4, we compare the total average cost over a month of three representative cases: two-timescale markets with 0.5M W h UPS, real-time market with 0.5M W h UPS, and real-time market without UPS. Although purchasing power from the real-time market has higher electricity price, the additional real-time procurement can help reduce 4.87% cost. The rationale is that CSP can purchase the exact amount of power from the two-timescale markets to avoid costs due to inaccurate procurement (extra actions of purchasing and charging/discharging). In addition, we can see that the benefit brought by energy storage is about 19.3% higher than that of the markets. In sum, with twotimescale power markets and different battery sizes, OnDPSS can reduce operational cost by 4.87% 24.17%. V. CONCLUSION This paper applied two-stage Lyapunov optimization to design an online control algorithm (OnDPSS) that best utilizes different characteristics of multiple energy sources in a complementary manner to deliver reliable energy to datacenters while minimizing DPSS operational cost, without requiring substantial statistics of system dynamics. Both mathematical analyses and trace-driven evaluations demonstrated the optimality and stability of OnDPSS. It can approach the optimal solution within a diminishing gap of O(1/V ), which is mainly Fig. 4. Total Cost ($) TwoMarket+NoUPS TwoMarket+0.25MWh UPS TwoMarket+0.5MWh UPS TwoMarket+1MWh UPS RealTime+NoUPS RealTime+0.5MWh UPS UPS Size Impact Grid Markets Impact Impact of battery capacity and grid markets structure in total cost. limited by the UPS battery size, multi-timescale grid markets and DPSS operation frequency. VI. ACKNOWLEDGEMENT The research was supported in part by a grant from The National Natural Science Foundation of China (NSFC) under grant No and No , by a grant from the Research Fund of Young Scholars for the Doctoral Program of Higher Education, Ministry of Education, China, under grant No , by the CHUTIAN Scholar Project of Hubei Province. REFERENCES [1] A. Qureshi, Power-Demand Routing in Massive Geo-Distributed Systems, Ph.D. dissertation, Massachusetts Institute of Technology, [2] How Clean is Your Cloud, [3] [4] M. He, S. Murugesan, and J. Zhang, Multiple Timescale Dispatch and Scheduling for Stochastic Reliability in Smart Grids with Wind Generation Integration, in Proc. of INFOCOM, Apr [5] S. Adlakha and A. Wierman, Energy Procurement Strategies in the Presence of Intermittent Sources, Technical report, [6] L. Jiang and S. Low, Multi-period Optimal Procurement and Demand Responses in the Presence of Uncrtain Supply, in Proc. of IEEE Conference on Decision and Control (CDC), [7] L. Huang, J. Walrand, and K. Ramchandran, Optimal Power Procurement and Demand Response with Quality-of-usage Guarantees, Arxiv preprint arxiv: , [8] P. Varaiya, F. Wu, and J. Bialek, Smart Operation of Smart Grid: Risklimiting Dispatch, in Proc. of the IEEE, vol. 99(1), pp , [9] S. Govindan, A. Sivasubramaniam, and B. Urgaonkar, Benefits and Limitations of Tapping into Stored Energy for Datacenters, in Proc. of ACM ISCA, Jun [10] R. Urgaonkar, B. Urgaonkar, M. Neely, and A. Sivasubramanian, Optimal Power Cost Management Using Stored Energy in Data Centers, in Proc. of ACM SIGMETRICS, Jun [11] C. Stewart and K. Shen, Some joules are more precious than others: Managing renewable energy in the datacenter, in Proc. of the Workshop on Power Aware Computing and Systems, [12] L. Huang, J. Walrand, and K. Ramchandran, Optimal Demand Response with Energy Storage Management, Arxiv preprint arxiv: , [13] Í. Goiri, K. Le, T. Nguyen, J. Guitart, J. Torres, and R. Bianchini, GreenHadoop: Leveraging Green Energy in Data-Processing Frameworks, in Proc. of EuroSys, Apr [14] Y. Zhang, Y. Wang, and X. Wang, GreenWare: Greening Cloud-Scale Data Centers to Maximize the Use of Renewable Energy, in Proc. of ACM Middleware, Dec [15] L. Georgiadis, M. Neely, M. Neely, and L. Tassiulas, Resource Allocation and Cross-layer Control in Wireless Networks. Now Pub, [16] Y. Yao, L. Huang, A. Sharma, L. Golubchik, M. Neely et al., Data Centers Power Reduction: A Two Time Scale Approach for Delay Tolerant Workloads, in Proc. of IEEE INFOCOM, Mar [17] FAQ.htm. [18] W. Deng, F. Liu, H. Jin, and X. Liao, Online Control of Datacenter Power Supply under Uncertain Demand and Renewable Energy, HUST Technical report, Sep [19] MIDC, [20] NYISO,