Capping the Brown Energy Consumption of Internet Services at Low Cost Kien T. Le Ricardo Bianchini Thu D. Nguyen Rutgers University Ozlem Bilgir Margaret Martonosi Princeton University
Energy Consumption of Data Centers Electricity consumption of US DCs Billion KWh/year 140 120 100 80 60 40 20 0 $7.4B $4.5B NA 2000 2006 2011 Equals consumption of transportation manufacturing industry in 2006 Est. 124.5 BKWh for 2011 Source: EPA 2006
Environmental Costs 100 % 80 % 60 % 40 % 20 % Electricity sources Others Renewables Nuclear Natural Gas Coal Carbon emissions of world-wide DCs MTT/year 120 115 110 105 35th 34th 0 % US World 100 Nigeria Data Centers Czech Rep Sources: DOE and Mankoff08
Capping Brown Energy Consumption Improving efficiency does not promote green energy or guarantee limits on brown energy Trend: Cap the brown energy consumption of large electricity consumers (data centers) Capping schemes Cap-and-trade: purchase carbon offsets Cap-and-pay: pay higher brown energy price Cap-as-target: pay more for neutrality
Capping Brown Energy Consumption (cont.) Real example: UK CRC Energy Efficiency Scheme Mandatory cap-and-trade scheme starting in April 2010 Organizations consuming 6 GWh/year Affecting 20,000 organizations Cap brown energy without degrading performance or excessively increasing costs and overheads? Our current focus: Multi-DC Internet services
Characteristics of Internet Services Company Servers Electricity Cost CO 2 (Tons) ebay 16K 0.6 10 5 MWh $3.7M 0.43 10 7 Akamai 40K 1.7 10 5 MWh $10M 1.2 10 7 Rackspace 50K 2 10 5 MWh $12M 1.4 10 7 Microsoft >200K > 6 10 5 MWh >$36M 4.3 10 7 Google >500K > 6.3 10 5 MWh >$38M 4.5 10 7 Source: Qureshi et al., SIGCOMM 09
Multi-DC Internet Services Front-end Data Center 3 Internet Data Center 2 Front-end Data Center 1
Across-DC Request Distribution Front-end DC mirror 3 Internet Front-end Typically, a request can be served by 2-3 mirror DCs DC mirror 2 DC mirror 1
Across-DC Request Distribution Front-end DC mirror 3 Internet Front-end Request distribution policy determines the DC to use DC mirror 2 DC mirror 1
Across-DC Request Distribution Front-end DC mirror 3 Internet Front-end Policy must account for potential increase in response time (SLA) DC mirror 2 DC mirror 1
Geographical Distribution of Data Centers DC Mirror 1 Large capacity Cheap brown energy Wind energy High latencies DC Mirror 2 Small capacity Costly brown energy No green energy Low latencies DC Mirror 3 Small capacity Costly brown energy Wind energy High latencies
Our Work Time Series Analysis Statistical Performance Data DC mirror 3 Workload Front-end DC mirror 2 Compute Yearly Mixes Request Distribution Minimize cost while satisfying SLA DC mirror 1
Cap-and-Trade Distribution Policy Goal: Compute f i s that minimize the overall energy cost Costs: Base energy + dynamic per-request energy On/off-peak electricity pricing Brown vs. green electricity pricing Purchase offsets if brown cap exhausted Constraints: DCs must not be overloaded SLA must not be violated
Approach: Optimization Optimization-Based Distribution Formulate an optimization problem Compute power mixes for a year Periodically compute distribution fractions Use simulated annealing (SA): Week-long load predictions, 4-hour epochs
Approach: Heuristics Heuristic-Based Distributions Simpler approach Forward client requests greedily based on cost and performance Cost-aware heuristic (CA): every hour; best cost-perf ratio from well-performing DCs first; later, lowest price Cost-unaware heuristic (CU): best performing DCs Common Power mixes computed with SA Communicate with DCs for server turn on/off
Evaluation Setup DCs: Washington, New Jersey, Switzerland Front-end: New Jersey Year-long trace of a commercial service Load prediction using ARIMA within 10% Real network latencies, brown & green energy prices (on/off-peak), carbon market prices Max 30% green energy; Cap: 75% of dynamic energy (enforced per year); SLA: 90% 500ms (per week)
Evaluation Methodology Methods: Simulations Real system to validate simulations SLAs are always satisfied Real experiments Based on HAProxy, added 3K lines Ran for 40 hours to validate Results within 6% of simulations
Optimization Beats Heuristics 120 % sa ca cu Normalized Cost and Brown Energy 100 % 80 % 60 % 40 % 20 % 61 94 100 100 100 95 0 % Energy Cost Brown Energy SA achieves 39% cost savings CA lowers costs only slightly
Meeting SLA 100 % Percent requests served within 500ms 90 % 80 % 70 % 60 % sa ca cu P 50 % 0 24 48 72 96 120 144 168 Hour SLA: 90% 500ms (per week) SLAs are satisfied; SA takes advantage of cheap energy
Comparing Caps Normalized Cost and Brown Energy 140 % 120 % 100 % 80 % 60 % 40 % 20 % 123 50% 110 75% 100% 100 100 76 70 0 % Energy Cost Brown Energy 75% cap lowers brown energy by 24% for a 10% cost increase
Conclusion Proposed request distribution framework for multi-data-center Internet services to deal with brown energy caps Proposed optimization and heuristic techniques for managing energy and cost Approach is effective at capping brown energy at low cost Optimization behaves better than heuristics Given expected brown energy caps, framework and policy should be useful in practice