Intelligent Placement of Datacenters for Internet Services

Transcription

1 Intelligent Placement of Datacenters for Internet Services Íñigo Goiri, Kien Le, Jordi Guitart, Jordi Torres, and Ricardo Bianchini Universitat Politécnica de Catalunya Barcelona Supercomputing Center Barcelona, Spain Department of Computer Science Rutgers University Piscataway, NJ USA Abstract Popular Internet services are hosted by multiple geographically distributed datacenters. The location of the datacenters has a direct impact on the services response times, capital and operational costs, and (indirect) carbon dioxide emissions. Selecting a location involves many important considerations, including its proximity to population centers, power plants, and network backbones; the source of the electricity in the region; the electricity, land, and water prices at the location; and the average temperatures at the location. As there can be many potential locations and many issues to consider for each of them, the selection process can be extremely involved and time-consuming. In this paper, we focus on the selection process and its automation. Specifically, we propose a framework that formalizes the process as a non-linear cost optimization problem, and approaches for solving the problem. Based on the framework, we characterize areas across the United States as potential locations for datacenters, and delve deeper into seven interesting locations. Using the framework and our solution approaches, we illustrate the selection tradeoffs by quantifying the minimum cost of (1) achieving different response times, availability levels, and consistency times, and (2) restricting services to green energy and chiller-less datacenters. Among other interesting results, we demonstrate that the intelligent placement of datacenters can save millions of dollars under a variety of conditions. We also demonstrate that the selection process is most efficient and accurate when it uses a novel combination of linear programming and simulated annealing. I. INTRODUCTION Today, popular Internet companies, such as Google, Yahoo, and Microsoft offer a range of services to millions of users every day. These services are hosted in datacenters that contain thousands of servers, as well as power delivery (and backup), cooling, and networking infrastructures. Because users demand high availability and low response times, each service is mirrored by multiple datacenters that are geographically distributed. Each datacenter is supposed to serve the requests of the users that are closest (in terms of network latency) to them. If this datacenter becomes unavailable or unreachable, these requests are forwarded to a mirror datacenter. Although the users experience is an overriding concern, Internet companies must also consider the enormous cost of provisioning and operating such a network of datacenters. The capital costs include land acquisition, datacenter construction, and bringing enough network bandwidth and electricity to the datacenter. The operational costs include electricity, water (for the cooling system), and system administration staff. Moreover, companies are starting to consider the environmental impact of their infrastructures. In particular, a datacenter may have a significant impact, depending on the type of electricity (renewable vs. CO 2 - intensive) it consumes. An interesting aspect of these response time, availability, cost, and environmental concerns is that many of them depend heavily on the specific locations of datacenters. For example, land, electricity, and water prices, as well as network latency, depend directly on location. Even more interestingly, the location-dependent costs are not always directly related. For example, land is cheap in a desert area but cooling is expensive. Given the pervasive impact of the datacenters locations, companies must intelligently select them. As there can be many potential locations for each datacenter and many issues to consider in evaluating them, the selection process can be involved and time-consuming. Unfortunately, because competition between services is fierce, no information is available in the public domain about how Internet companies actually select locations for their datacenters. Thus, in this paper we study the selection process, while fully characterizing the different parts of the US as potential locations for datacenters. First, we propose a framework for selection that includes parameters representing all aspects of datacenter costs, response times, data consistency, and availability. The framework allows us to define the selection process as a non-linear cost minimization problem with response time, consistency, and availability as constraints. Second, we propose approaches for solving the problem efficiently. Some of our approaches transform the problem into a collection of linear problems, whereas others use linear programming only in an auxiliary role. Third, we characterize areas across the entire US according to the parameters of our framework. The characterization includes each area s population, average temperature, electricity prices, average datacenter Power Usage Efficiency (or simply PUE), and proximity to network backbones and power sources (plants or transmission lines), and the CO 2 intensity of the area s power generation. We then delve deeper into seven interesting areas as potential locations

2 for a datacenter: Austin (Texas), Bismarck (North Dakota), Los Angeles (California), New York (New York), Orlando (Florida), Seattle (Washington), and St. Louis (Missouri). Based on our framework, solution approaches, and characterization data, we built a tool for selecting datacenter locations. The tool is extensible so that new parameters and constraints can be added. The tool can be used to select locations for an entire network of datacenters or to extend an existing network with one or more datacenters. 1 Our evaluation uses the tool to compare the efficiency and accuracy of approaches for solving the optimization problem. We also identify the most important optimization criteria, and illustrate the selection tradeoffs by quantifying the minimum cost of (1) achieving different response times, availability levels, and consistency times, and (2) restricting services to green energy and chiller-less datacenters. Our results demonstrate that the intelligent placement of datacenters can save millions of dollars under a variety of conditions. They also demonstrate that the selection process is most efficient and accurate when it uses a novel combination of linear programming and simulated annealing. Finally, among other interesting results, we show that the cost of a network of datacenters almost doubles when the maximum acceptable response time is decreased from 50ms to 35ms. In summary, our contributions are: We propose a framework and optimization problem for selecting datacenter locations; We propose solution approaches for the problem; Using extensive real data, we characterize areas across the US as potential locations for datacenters; and We answer many interesting quantification and tradeoff questions about datacenter networks. As far as we know, no previous work has considered the selection of locations for the datacenters of an Internet service in detail. The remainder of this paper is organized as follows. The next section describes our framework, optimization problem, and solution approaches. Section III describes our tool, highlighting how we instantiate the components of the framework using real cost and location data. Section IV characterizes the US using our real data and optimization machinery. Section V evaluates our solution approaches and assess the importance of the different optimization criteria. Section VI answers questions about datacenter network design and costs. Section VII presents the related work. Finally, Section VIII draws our conclusions. II. FRAMEWORK FOR PLACEMENT Our main goal is to efficiently select locations for one or more datacenters, so that the overall cost is minimized and the service s response time, consistency, and availability 1 More information about our tool, including a short video illustrating its use, can be found at requirements are respected. To that end, the next subsection defines the most important parameters in the selection process. Based on these parameters, we then formulate a cost model and optimization problem. The last subsection proposes approaches for solving the problem. A. Parameters Table I lists the entire set of parameters in our framework. They range from inputs provided by the user (MaxS, ratioserveruser, MAXLAT, MAXDELAY, and MINAVAIL) to parameters that we seek to instantiate via optimization (S d, P d c, SB d, and P B d c ). Among the more interesting parameters are those related to costs, temperatures, power and energy consumptions. Next, we discuss those related to costs in detail. Costs. The overall cost of a network of datacenters can be broken down into capital (CAPEX) and operational (OPEX) components. The CAPEX costs are those investments made upfront and depreciated over the lifetime of the datacenters. CAPEX can be further divided into capital costs that are independent of the number of servers to be hosted (CAP ind), those that do depend on the maximum number of servers that can be hosted (CAP max), and those that depend on the actual number of hosted servers (CAP act). (The maximum and actual number of servers may differ, because a datacenter may not be fully populated right from the start of its operation or because some servers are kept off during operation.) The CAP ind costs relate to bringing electricity and external networking to the datacenters. (Although the amount of electricity and external bandwidth depends on the number of servers, the base cost of laying out any transmission line or optical fiber dominates.) These costs vary according to location. They can be estimated from the distance between the location and (1) the closest transmission line or power plant; and (2) the closest network backbone. The CAP max costs relate to land acquisition, datacenter construction, and purchasing and installing the power delivery, backup, and cooling infrastructures. The land price varies according to location, whereas the other costs do not to a first approximation. All of these costs depend on the level of redundancy that will be built into each datacenter. The construction cost is typically estimated as a function of the maximum power to be consumed by the datacenter. This maximum power is that required by the maximum number of servers (and networking gear) when running at 100% utilization times the maximum expected PUE of the datacenter. The PUE is computed by dividing the overall power consumption by the power consumption of the computational equipment. The PUE is higher when temperature and/or humidity are high, since cooling consumes more energy under those conditions. CAP act accounts for the expenses with purchasing the

3 Symbol Meaning Unit Symbol Meaning Unit S d Max #servers in DC d serv SB d Is a DC placed in location d? bool P d c Actual #servers of DC d providing c serv P B d c Is a DC d providing c? bool MaxS Max #servers across all DCs serv MAXLAT Max network latency ms MINAV AIL Min availability % MAXDELAY Max consistency delay ms latency(p 1, p 2) Latency between p 1 and p 2 ms distance(p 1, p 2) Distance between p 1 and p 2 mile closerp ower(d) Closer power source (grid or plant) coord closernetwork(d) Closer network backbone coord emissions(d) CO 2 emissions at location d g/kwh population(c) Population of center c user priceland(d) Land price at location d $/sf pricew ater(d) Water price at location d $/gal priceenergy(d) Energy price at location d $/Wh avgt emp(d) Avg temperature at location d C maxt emp(d) Max temperature at location d C priceservernet Price including internal networking $/serv powerserver(u) Power at u util. (incl. network) W/serv ratioserveruser Servers required per user serv/user servers(c) population(c) ratioserveruser serv pricelinenet Price to layout optical fiber $/mile pricelinep ower Price to layout power line $/mile pricebuild(power) Price of building a DC of power W $/W puedc(t) PUE of DC at temperature t landdc Land required sf/w waterdc Water required gal/wh maintdc Cost to maintain DC $/W adminserver Administrators required adm/serv adminsalary Administrator salary $/adm networkserver External net bandwidth required Mbps/serv pricenetwork Price of net bandwidth $/Mbps T Operation period h maxp ower(d) powerserver(1) pue(maxt emp(d)) energy(u, d) T powerserver(u) pue(avgt emp(d)) buildcost(d) maxp ower(d) pricebuild(maxp ower(d) S d ) landcost(d) maxp ower(d) landdc priceland(d) CAP act priceservernet CAP max(d) landcost(d) + buildcost(d) CAP ind(d) pricelinep ower distance(closerp ower(d), d) + pricelinenet distance(closernetwork(d), d) OP act(d) maxp ower(d) maintdc + adminserver adminsalary + networkserver pricenetwork OP utl(u, d) energy(u, d) priceenergy(d) + energy(u, d) waterdc pricew ater(d) Table I SUMMARY OF FRAMEWORK PARAMETERS. EACH LOCATION d BELONGS TO THE SET D OF POSSIBLE LOCATIONS, WHEREAS EACH POPULATION CENTER c BELONGS TO THE SET C OF POPULATION CENTERS. servers and (internal) networking gear that will be hosted by the datacenters. These costs are proportional to the actual number of hosted servers and do not depend on location. The OPEX costs are those incurred during the operation of the datacenters. OPEX can be divided into costs relating to the actual number of hosted servers (OP act) and those that also depend on the utilization of those servers (OP utl). OP act relates to the maintenance and administration of the equipment, as well as their external network bandwidth use. Maintenance and administration costs are dominated by staff compensation. OP act costs depend on location. OP utl represents the electricity and water costs involved in running the servers at a certain level of utilization. These costs can be computed based on the servers energy proportionality [1] and average utilization (typically around 30% [1]), the average PUE, and the electricity and water prices. These costs vary with location. Finally, lower taxes and one-time incentives are another important component of the cost of a datacenter. For example, some states lower taxes on datacenters, as they generate employment and wealth around them. This component depends on the nature of the savings and applies to each cost component in a different way. Although we do not consider this component further, it is easy to include it in our framework. Response time. As mentioned above, Internet services must exhibit low response times to remain competitive. Thus, in selecting locations, it is critical to model the network latency between the service s potential users and the potential locations. We define latency(c,d) to be the oneway network latency between a population center c and a potential location d. In addition, we define Pc d to be the number of servers at location d that serve requests from center c, and servers(c) to be the number of servers required by center c. In more detail, servers(c) is the average number of users of the service in population center c (e.g., 30% of the population of c) divided by the average number of users that can be accommodated by a server. Consistency delay. Because services must keep working seamlessly even when a datacenter becomes unreachable or unavailable, the datacenters that provide a service must be mirrors of each other. This implies that changes to persistent state must be dynamically propagated to all the mirrors. The consistency delay refers to the time required for state changes to reach all mirrors. To model consistency, we define latency(d 1,d 2 ) to be the one-way network latency between potential locations d 1 and d 2. We model the consistency delay as the maximum network latency between any two mirror locations. Availability. The availability of a service depends on the availability of the network of datacenters that host the service. We model the availability of such a network as: n 1 ( ) n Availability = a n i (1 a) i (1) i i=0 where n is the number of datacenters and a is the availability of each datacenter. This model has been used in multiple previous reliability and availability works, e.g. [2], [3]. The availability of each datacenter depends on the level

4 of redundancy in its design. Industry commonly classifies datacenters into tiers [4], where each tier implies a different redundancy level and expected availability. At one extreme, Tier I datacenters have a single path for power and cooling distribution. At the other extreme, Tier IV datacenters have two active power and cooling distribution paths, with redundant components in each path. Existing Tier I datacenters have been found to achieve an availability of 99.67%, whereas Tier II datacenters achieve 99.74%, Tier III datacenters achieve 99.98%, and Tier IV datacenters achieve % [4]. CO 2 emissions. The type of electricity consumed at each potential location determines the resulting carbon emissions of the datacenter. (Obviously, the construction and operation of datacenters involves other sources of carbon emissions. For simplicity, we do not model them.) We define emissions(d) as the carbon emissions in grams/kwh of the electricity generation at location d. B. Formulating the optimization problem Using the parameters of our framework, we now formally define the cost model and optimization problem. The problem setup consists of an Internet company that seeks to select locations for a set of datacenters out of a set of potential locations for them (D). The company also seeks to offer its services to a set of population centers (C), so each datacenter needs to be sized accordingly. The optimization goal is to minimize the overall cost of the datacenter network, while respecting network latency, consistency delay, and/or availability constraints. The inputs to the optimization are (1) the maximum number of servers to be hosted across the network of datacenters (MaxS); (2) the expected average utilization for the servers in use (Util); (3) the number of users that each server can accommodate; (4) the amount of redundancy that will be built into each datacenter; (5) the maximum network latency (MAXLAT), maximum consistency delay (MAXDELAY), and/or minimum availability (MINAVAIL) required by the network of datacenters; (6) the CAPEX and OPEX costs for each location d D; (7) the one-way network latency between any population center c C and each location d; and (8) the one-way network latency between any two locations d 1 and d 2. The outputs of the optimization are the optimal cost, the maximum number of servers at each location d (S d ), and the number of servers that service population center c at location d (Pc d ). Equation 2 in Figure 1 defines the cost we seek to minimize (TotalCost), where SB d and P Bc d are booleans. SB d represents the placement of a datacenter at location d, i.e. it is defined as 1 if S d > 0, and 0 if S d 0. Similarly, P Bc d represents the use of servers at location d to serve users from a population center c, i.e. it is defined as 1 if Pc d > 0, and 0 if Pc d 0. Recall that CAP ind is the CAPEX cost that is independent of the number of servers; CAP max is the CAPEX cost per server assuming the maximum number of servers; CAP act is the CAPEX cost per hosted server; OP act is the OPEX cost per hosted server; and OP utl is the utilization-dependent OPEX cost per hosted server. All costs assume 12 years of lifetime for datacenters and 4 years for servers. In addition, our modeling assumes that the CAPEX costs already embody the depreciation costs and any interest payments they may incur. TotalCost should be minimized under the constraints that follow the equation in the figure. The constraints include the network latency, consistency delay, and availability requirements of the service. In reality, the availability constraint is more complex than in the figure. In particular, in a network with at least 2 datacenters and S servers, we also ensure that the failure of 1 datacenter will leave S/2 servers available to handle the load. Similarly, in a network with at least 3 datacenters, we ensure that the failure of 2 datacenters will leave S/3 servers available. Existing datacenters. We can use a small variation of the problem in Figure 1 for adding datacenters to an existing network of datacenters. In fact, this type of network extension is likely to be the most common scenario for real Internet companies. Companies may want to extend their networks to provide more services, serve more users, reduce latency, reduce consistency delay, or increase availability. The variation adds new constraints specifying the number of servers at each datacenter that has already been allocated. With these constraints, we can generate the location of the new datacenters and their size. C. Solution approaches The problem described in Figure 1 can be time-consuming to solve for two reasons: (1) it is non-linear, since SB, PB, and CAP max are non-linear functions (CAP max has different values per MW depending on the size of the datacenter); and (2) there may be a large number of potential locations to evaluate. Unfortunately, the fact that the problem is non-linear means that it is not directly solvable by linear programming (LP) solvers, which are very fast. If an LP solver is to be used at all, we must eliminate the nonlinearities (perhaps at the cost of accuracy). Next, we discuss approaches that use LP to different extents. Simple linear programming (LP0). The first solution approach reformulates the optimization problem to remove variables S d and Pc d, and use a linear version of CAP max (called LCAP max). The removal of S d and Pc d requires stricter constraints on the placement of servers for each population center (constraints 5 and 6). The reformulated problem appears in Figure 2. We solve the reformulated problem to find SB d and P Bc d directly. The actual number of servers at each datacenter is

5 T otalcost = SB d CAP ind(d) + S d CAP max(d) + Pc d (CAP act + OP act(d) + OP utl(util, d)) (2) d D d D c C d D 1. d c P Bc d latency(c, d) MAXLAT i.e., no user should experience higher latency than MAXLAT. 2. d 1, d 2 SB d1 SB d2 latency(d 1, d 2) MAXDELAY i.e., consistency should take no longer than MAXDELAY. 3. Availability MINAVAIL i.e., availability (defined in Equation 1) must be at least MINAVAIL. 4. servers(c) MaxS i.e., the total number of servers is no greater than MaxS. c 5. c P d d c = servers(c) i.e., must provision enough servers for every population center. 6. d P d c c S d i.e., no datacenter can host more servers than its max capacity. Figure 1. We solve the optimization problem to find S d and P d c (and consequently SB d and P B d c ) for every population center c and potential datacenter location d. CAP ind = CAPEX independent of the number of servers; CAP max = CAPEX per server assuming maximum configuration; CAP act = CAPEX per hosted server; OP act = OPEX per hosted server; OP utl = utilization-dependent OPEX per server. All costs assume a 12-year lifetime for datacenters and a 4-year lifetime for servers. ApproxCost = SB d CAP ind(d)+ P Bc d servers(c) (LCAP max(d) + CAP act + OP act(d) + OP utl(util, d)) d D c C d D 1-4. Same as in Figure c d D P Bd c = 1 i.e., each population center is served by a single datacenter. 6. c d P B d c SB d i.e., a datacenter must exist to serve a population center. Figure 2. LP0 solves the optimization problem to find SB d and P B d c for every population center c and potential datacenter location d. LCAP max is a linear version of CAP max. The first 4 constraints are the same as those in Figure 1. Again, all costs assume a 12-year lifetime for datacenters and a 4-year lifetime for servers. (3) then c d P Bd c server(c). To retain linearity, this value is also assumed to be the maximum number of servers of each datacenter. In addition, LCAP max assumes the same cost per MW for large datacenters as for small datacenters. After the problem is solved, we proportionally set the maximum number of servers at each datacenter, so that the sum of all maximum numbers of servers is equal to MaxS. With the maximum number of servers defined for each datacenter, we compute its CAP max using the proper costs per MW. Finally, we use the original function (Equation 2) to compute the cost of the network of datacenters. Due to its simplifications and restrictiveness, this approach may produce higher total cost for a datacenter network than the other approaches. Pre-set linear programming (LP1). Another approach for linearizing the problem is to pre-set SB d and remove P Bc d. Pre-setting is effected by creating additional equality constraints to the problem. Instead of using SB d as a boolean, we pre-set it to 0 (meaning no datacenter at location d), 1 (small datacenter at location d), or 2 (large datacenter at location d). Using these values, we can apply the proper CAP max. The removal of P Bc d only affects the maximum network latency constraint. Instead of using that constraint, we check the latency between every datacenter d and population center c. If the latency is larger than MAXLAT, we pre-set Pc d to 0. This approach allows us to solve the problem to find S d and the values for Pc d that were not pre-set. As it requires a previously selected set of datacenters, this approach cannot be used by itself. Nevertheless, it will be used by the next solution approaches to calculate the server distribution and the cost for a given set of datacenters. Brute force (Brute). This approach generates all the possible combinations of datacenters in every tentative location and tests each of them using the LP1 approach. It ranks each of the placements and returns the best one. Since this solution approach is exhaustive, it obtains the optimal result at the cost of extremely long execution times. Heuristic based on LP (Heuristic). This approach comprises three steps. The first step uses LP0 to generate a set of M1 (M1 = 10 by default) datacenter networks for each number of datacenters ranging from 1 to D. We limit the number of datacenters using an additional constraint ( d SB d = NumDC) that is added to the problem after each set of M1 configurations is produced. As we generate each set, every time a configuration is produced, a new constraint ( SB d =1 P reviousconfig SB d < NumDC) is added to the problem that prevents the next execution of LP0 from selecting the same datacenter locations again. The first step leads to M 1 D configurations, each of which includes the datacenter locations, the maximum numbers of servers, and the actual numbers of servers. Since these LP0-derived configurations may be sub-optimal, the second step uses their SB d and P Bc d values to drive LP1. From the M1 D LP1 results, we create a ranking of configurations in increasing order of cost. Out of this ranking, we take the M2 (M2 = 10 by default) configurations with the best total costs. As some important combinations of locations may not have been considered by LP0 (and consequently LP1), the

6 Figure 3. Potential datacenter locations in the US using a grid. Population centers and network backbones are also shown. third step selects the most popular locations from the shorter ranked list of configurations and runs brute force on them. Simulated annealing plus LP1 (SA+LP1). This approach applies Simulated Annealing (SA) [5] and LP1 to solve the problem in Figure 1. SA is a generic probabilistic meta-heuristic for non-linear optimization problems. In our implementation, each candidate solution is the SB d and P B d c values for each potential location d and population center c. The optimization starts out with a configuration that fulfills the constraints and has one datacenter in each available location. From this point, in each iteration of the algorithm, SA evaluates one of the neighboring configurations of the current candidate solution using LP1. For example, a neighbor configuration may not include one of the datacenters in the current solution or it may include one in a new location. In each iteration, SA tries to find a lower cost neighbor to use as the next candidate solution, but may also select a higher cost neighbor (with a decreasing probability). If there are no cost reductions for a given number of iterations, the algorithm returns the best candidate solution considered during the optimization. Optimized SA+LP1 (OSA+LP1). SA+LP1 can be optimized by adjusting the results produced by LP1. Specifically, when LP1 assigns no servers to a datacenter that is part of a configuration, the datacenter can be removed from it. OSA+LP1 then re-evaluates the modified configuration to get a new (lower) total cost. This approach speeds up the optimization process because it drives toward the lowest cost configuration faster. III. PLACEMENT TOOL AND INPUT DATA We created a datacenter placement tool based on the framework and optimization machinery described above. Our tool takes as input the parameters that define the user s desired datacenter network: M axs, 1/ratioServerU ser, MAXLAT, MAXDELAY, and MINAV AIL. In addition, the tool allows the user to define the area of interest, the granularity of the potential datacenter locations, and the location of the company s existing datacenters (if any). The default area of interest is the entire US, but any geographical area can be used as long as we can instantiate the framework parameters for it. The granularity is specified as an n n grid of tiles. For example, Figure 3 shows the US split into a grid. Note that some of the tiles are located in areas where a datacenter cannot be placed, such as an ocean, a lake, Canada, or Mexico. Overall, we get 253 potential locations with this grid size. These locations form the D set. The set of users to be served is assumed to be a (configurable) fraction of the population of the main metropolitan centers within the area of interest. Figure 3 illustrates the 66 metropolitan centers in the US using squares of sizes proportional to their populations. These centers form the C set. Overall, the total population of these centers is 174 million people. We obtained the information about each location in D from Internet services that provide the coordinates (latitude and longitude) of different places and the name of each place 2. With the coordinate information, we instantiate framework parameters distance(p 1, p 2 ). We obtained the population information about the centers in C from the US census 3. With the population information, we instantiate parameters population(c). Next, we describe how we instantiate the other location-dependent and datacenter parameters of our framework and tool. A. Location-dependent data Network backbones. We extracted the backbone information from the topology of the different ISP backbones 4. Figure 3 illustrates them with straight lines. In addition, we obtained the shortest path between any two points in the network using Dijkstra s algorithm [6]. Following a model based on the number of hops and the distance [7], [8], we estimate the latency for every path. Specifically, the model assumes an average delay of 8 milliseconds per hop and a propagation speed equivalent to the speed of light on fiber (around 193,121km/s). This model fits the latency data available from the ISPs 5. With these information, we instantiate parameters latency(p 1, p 2 ) and closestnetwork(d). Power plants, transmission lines, and CO 2 emissions. We extracted the power plant location and type information from the Department of Energy (DOE) [9], whereas information about the main power transmission lines came from National Public Radio 6. Figure 4 illustrates the locations of the major power plants in the US, as well as the major transmission lines. We obtained information about the CO 2 emissions resulting from electricity generation from [10]. Using these data, we instantiate closestp ower(d) and emissions(d) delay.html 6

7 Figure 6. PUE o Temperature ( C) Datacenter PUE, as a function of outside temperature. Figure 4. Energy prices (darker is more expensive), power plants, and transmission lines across the US. Figure 5. Average temperatures across the US (darker is warmer), as well as population centers. Electricity, land, water, and temperature. We obtained the price of electricity from the DOE 7. For example, the electricity price is low (around $0.04 per KWh) in Washington State, whereas it is high (around $0.14 per KWh) in Rhode Island. Figure 4 plots these data. With electricity price information, we instantiate parameters priceenergy(d). We obtained the price of the land using the methodology presented in [11], which gets its data from a real-state portal 8 and calculates an average price for each location. We use data about industrial land with more than 10K square feet (SF), which is suitable for a datacenter. These data instantiate parameters priceland(d). The amount of water that is used for cooling a datacenter depends on the energy consumption and indirectly on the average temperature at the location. To obtain the temperature information, we used a weather portal with historical data 9. Figure 5 illustrates the average temperatures in the US. These information allow us to instantiate avgt emp(d) and maxt emp(d). We obtained the price of water from a study of water rates and pricing in different cities [12]. Missing data. Depending on the grid size, our tool may not have all the important data for every potential location. In these cases, the tool uses information from the closest neighboring location for which it has the needed data. If there is a tie in the distance, it chooses one of the neighboring locations randomly. B. Datacenter characteristics Datacenter size, cooling, and PUEs. Current datacenters are commonly measured by the maximum power they are designed to consume. The maximum power is a function of the maximum PUE that can be expected, which in turn depends on the type of cooling, the power delivery losses, and the maximum temperature at the location. Regarding the cooling system, we assume that the datacenters are cooled by Computer Room Air Conditioners (CRACs) and water chillers. The CRACs are always running, but the chiller is only activated when a threshold outside temperature is crossed (20 C). Thus, energy consumption increases with outside temperature. To complete the computation of the PUEs, we assume that the power delivery losses are 8% by default. Taking all of these factors into account, Figure 6 illustrates our modeling of the PUE as a function of outside temperature. Connection costs. Since each datacenter will not necessarily be located near a power plant or existing transmission line, there may be costs associated with laying out lines to bring enough electricity to it. The same can be said about laying out optical fiber to reach an Internet backbone. Our default values for these costs are $500K per mile of transmission line and $480K per mile of fiber laid out [13]. The amortization period for this infrastructure is 12 years. We assume that the amount of external network bandwidth required by a datacenter depends linearly on the number of servers it hosts. Our default amount is 1Mbps per server. A high bandwidth link to a backbone typically costs between $1 and $2 per Mbps 10 ; our default cost is $1 per Mbps. Building costs. The cost of building a datacenter, including purchasing and installing its cooling and power delivery infrastructures is typically computed as a function of its maximum power [14]. In particular, small datacenters ( 10MW) are more costly to build than large ones (> 10MW). The specific cost per Watt depends on the level of availability desired for the datacenter [14]. For example, building a small Tier I datacenter costs $10 per Watt, whereas a large one costs $8 per Watt. The numbers for the other tiers are $11 and $8.8 for Tier II, $20 and $16 for Tier III, and $22 and $17.6 for Tier IV. This cost is amortized for 12 years. In all our experiments below, we assume new datacenters close to Tier III with $15 (small) and $12 (large) per Watt

8 and a typical availability of %, as suggested by [1]. Land cost. The amount of real-estate required by a datacenter is also typically computed based on its maximum power. Current datacenters occupy around 6K SF per Megawatt [14]. Using this size ratio and the price of land, we compute the total land cost. Water cost. According to [15], a datacenter of 15 Megawatts with the type of cooling we model uses around 360K gallons of water per day. Using the same ratio, we assume that a datacenter consumes 24K gallons of water per MW per day. From this water consumption and the water price, we compute the total water cost. Servers and internal networking costs. Our default servers are Dell PowerEdge R610 with 4 processor cores running at 2.66GHz with 6GBytes of RAM. These servers consume a maximum of 260 Watts and an average of 200 Watts at a 30% utilization (a common average utilization in Internet service datacenters [1]). The cost of each of these servers is around $2000, which is amortized over 4 years (i.e., each of them costs $42 per month). Our default interconnect is the 40-port Cisco Nexus 5020, which consumes 480 Watts. In our datacenter designs, each switch connects 32 servers; the switches also connect to each other hierarchically to reach all servers in the datacenter. The cost of each switch is $20K, which is amortized over 4 years. Staff costs. Any datacenter needs personnel to operate and maintain it. Following [1], our default for maintenance costs is $0.05 per Watt per month. In addition, our default operation costs assume that each administrator can manage 1K servers for an average salary of $100K per year. IV. EXPLORING THE TOOL In this section, we use our tool and input data to characterize seven interesting locations in the United States. Then, we present a case study in which we place a network of datacenters across the country. A. Characterizing locations To study the influence of the location on the cost of a datacenter, we have chosen seven representative locations with different characteristics: Austin, Bismarck, Los Angeles, New York, Orlando, Seattle, and St. Louis. Our study consists of placing a datacenter with 60K servers in each location, and assessing the components of the overall cost. The placement area is a square of 200 miles around these cities, which is divided in 9 tentative placement areas. Our optimization finds the most cost-effective of these areas. Figure 7 shows the contribution of each component to the overall cost of the datacenter per month. As it does not depend on the location, the cost with servers and internal networking is always the same: $3.3M per month. The other components are affected by location-dependent parameters, Cost (in million dollars) Austin Bismarck Los Angeles New York Orlando Seattle St. Louis Figure 7. Datacenter cost depending on location. Networking Staff Water Energy Connection Building Land Servers Land Energy Water CO 2 Location PUE ($/SF) ($/kwh) ( /gal) (g/kwh) Austin Bismarck Los Angeles New York Orlando Seattle St. Louis Table II LOCATION CHARACTERISTICS. such as electricity price, water price, land price, distance to infrastructure, and weather. Table II summarizes some of the main characteristics of the seven locations. From this figure and table, we make several observations: 1. Austin. Despite the fact that water is expensive in Texas, it does not represent a significant cost for a datacenter. Nevertheless, the arid weather in the region produces a relatively high average PUE of Bismarck. The Midwestern region is far from the existing network infrastructure. As a result, the cost of providing network connectivity to a datacenter in Bismarck is high. Moreover, the CO 2 emissions in that region are relatively high, as most of its electricity is coal-generated. 3. Los Angeles. Temperatures in Southern California are high, which implies a high average PUE. Land prices in this region are also high. These two factors make this location the most expensive to place a datacenter. 4. New York. Land in this region is the most expensive around the country. Electricity is also expensive. Again, these factors make the New York area an inappropriate location for a datacenter. 5. Orlando. Florida is the warmest state in the country, producing the highest average PUE of The PUE affects many costs, including building and land costs. Moreover, such a high PUE implies an electricity consumption that is unnecessarily high. 6. Seattle. Energy is inexpensive in Washington State, leading to one of the lowest electricity costs in the country. As most of the electricity in this region is produced by hydroelectric plants, the CO 2 emissions are low.

9 Cost (in million dollars) Figure Cost per month distribution in 500 locations of the US. 7. St. Louis. This location is in the center of the country; users in any part of the country can reach it in less than 75ms. In terms of costs, the St. Louis region exhibits low electricity, land, and water prices. However, its PUE and CO 2 emissions are relatively high. To get a deeper understanding of this datacenter s cost, we detail the most relevant aspects of the placement in St. Louis. We start with those aspects that affect the maximum power consumed by the datacenter. The average and maximum temperatures are around 13 C and 26 C, respectively. These temperatures lead to average and maximum PUEs of 1.32 and 1.6, respectively, As this datacenter hosts 60K servers, operating them at maximum utilization during the summer would produce a maximum power of 26.4 MW. For this maximum power, the building cost is $2.2M per month and the space required is around 158K SF (more than 2 and a half football fields). To connect the datacenter to the power and the network infrastructures, 60 miles of transmission lines and optical fiber would be required. These connections would cost $208K per month. In terms of operational costs, a datacenter in this region would use an average of MWh of energy per month. This consumption implies a cost of $573K per month and translates into the emission of 9876 tones of CO 2 per month (80% of the energy generated in Missouri comes from coal and 6% from natural gas). In addition, the cooling system of a datacenter of 26.4 MW consumes more than 12M gallons of water per month, translating into a cost of $25K per month. Finally, the labor costs of the datacenter would amount to $1.8M per month. The final cost of this data center is $8.22M per month, i.e. more than $98M per year or $1183M during the whole datacenter lifetime of 12 years. Broadening the scope. Still placing a single datacenter with 60K servers, we now consider the corresponding costs across the country. Figure 8 presents the distribution of the cost assuming 500 potential locations (a grid). The distribution shows that there are a few locations where the datacenter can be cheaper (around $7.35M), whereas others lead to much higher costs (around $9.84M). This observation highlights the importance of properly selecting locations for datacenter networks. A B C 2.42 MW MW 10.0 MW Cost (in million dollars) Figure 9. Networking Staff Water Energy Connection Building Land Servers Cost of a network of datacenters for 60K servers. SA OSA Grid D LP0 Heur. +LP1 +LP1 Brute Table III OVERALL COSTS (IN $M PER MONTH) OF SOLUTION APPROACHES. B. Case study: Placing a datacenter network As a complete example of the use of our tool, we compute the total cost of placing 60K servers, so that any user can reach the closest datacenter in less than 60ms, the maximum consistency delay is 85ms, and the minimum availability is 5 nines. Figure 9 presents the results. Our tool placed 3 datacenters (A, B, and C) close to Seattle, St. Louis, and Oklahoma City. These datacenters host 31789, 22712, and 5501 servers, respectively. The figure shows the total costs per month for the three datacenters. V. EVALUATING OUR SOLUTION APPROACHES To evaluate our solution approaches, we assess the running time and overall cost they produce, as a function of the number of potential locations. Our study assumes that the goal is to place 60K servers with a maximum latency of 60ms, a maximum consistency delay of 85ms, and a minimum required availability of 5 nines. Each experiment is performed on an 8-core 2.4GHz machine with 8GB of memory. Our tool fully exploits the 8 cores by running multiple solution threads in parallel. Table III shows the overall cost achieved by each solution approach. To assess the quality of the results, we compare them to the Brute approach, which computes the optimal cost. However, running Brute to completion is infeasible, so we stopped each of its executions after 3 days enough time to explore all possible placements of up to 5 datacenters. Beyond this point, better solutions are highly unlikely to exist. The colors in the figure represent how close each result is to Brute; darker colors mean worse solutions. Recall that

10 Optimizing time (seconds) Solution quality (%) Number of tentative locations Figure 11. Figure 10. Running times of solution approaches. Brute Heuristic SA+LP1 OSA+LP1 LP0 OSA+LP1 SA+LP Time (seconds) Solution quality over time using SA+LP1 and OSA+LP1. we cannot use LP1 in isolation, so we do not include it in this comparison. We observe that OSA+LP1 finds the optimal solution in all cases, whereas SA+LP1 only does so in two cases. The reason is that Simulated Annealing is more likely to get stuck in local minima when the search space is not pruned. We also see that Heuristic exhibits good behavior in most cases. However, this approach does not find the optimal results, as it leverages LP0 heavily. Finally, we find that LP0 exhibits the worst behavior. Due to its simplifications, LP0 is inflexible in terms of the assignment of user load to datacenters, and does not consider the non-linearity of building costs. Figure 10 shows the approaches running times. The times for Brute were extrapolated from the 3-day runs mentioned above, whereas the times for the other approaches were measured from complete executions. Note that the Y-axis is in log scale. From this figure, we can observe that all approaches scale linearly with the number of locations (quadratically with the dimensions of the grid), but with different slopes. LP0 exhibits the lowest running times, as it relies solely on linear programming with a small number of variables. At the other extreme, Brute exhibits very high running times. In fact, this approach takes multiple hours for as few locations as 18. As the other approaches explore much smaller spaces, they execute at least 3 orders of magnitude faster than Brute. Heuristic is the slowest of these approaches, whereas OSA+LP1 is the fastest. Overall, OSA+LP1 is 1-2 orders of magnitude faster than Heuristic and at least 5 orders of magnitude faster than Brute for nontrivial problem sizes. Figure 11 illustrates how quickly OSA+LP1 converges to its results compared to SA+LP1, for a scenario in which both approaches reached the same solution. It is clear from the Cost (in million dollars) Maximum latency (milliseconds) Figure 12. Cost of a datacenter network, as a function of max latency. Availability #9s Tier I Tier II Tier III Tier IV 99% % % % % Table IV NUMBER OF DATACENTERS REQUIRED FOR A GIVEN AVAILABILITY. figure that pruning the search spaces allows OSA+LP1 much faster convergence. Heuristic converges more slowly than these two approaches, as it has to wait until LP0 generates all the tentative locations to finally evaluate them using LP1. In contrast, the quality of the Brute solution over time does not exhibit a monotonic behavior, as it tries every possible solution in turn without regard for how good they are. Overall, it is clear that OSA+LP1 provides the best tradeoff between running time and search quality; it achieves optimal results with the second lowest execution times. VI. EXPLORING DATACENTER PLACEMENT TRADEOFFS We now use our tool (OSA+LP1) to answer the following interesting questions about datacenter placement and design: How much does lower latency cost? How much does higher availability cost? How much does faster consistency cost? How much does a green datacenter network cost? How much does a chiller-less datacenter network cost? Throughout the section, unless otherwise stated, we place a network of datacenters for 60K servers using a grid (253 potential locations). The default constraints are a maximum latency of 60ms, a maximum consistency delay of 85ms, and a minimum availability of 5 nines. A. Latency Figure 12 presents the total cost per month, as a function of the maximum latency constraint. We can see that required maximum latencies greater than 70ms all lead to the same cost, around $7.8M per month. Lower latencies require more datacenters and higher costs. A latency of 50ms strikes the best compromise between latency and cost; cost increases are steep for lower latencies. For 33ms, the minimum latency that can be achieved with this grid size, the total cost is $15.8M per month (almost twice the cost for 50ms).