Alejandro D. Domínguez-García, George Gross, and Philip T. Krein

nalysis of Design lternatives for Supplying Electric Power to Mission-Critical Loads in Data Centers: Reliability, Environmental, and Efficiency spects lejandro D. Domínguez-García, George Gross, and Philip T. Krein Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, Urbana, IL 61801, US E-mail: {aledan, gross, krein}@illinois.edu EXECUTIVE SUMMRY In this report, we analyze the reliability, the CO 2 emission impacts, and the overall efficiency of three faulttolerant power supply topologies for mission-critical IT loads that eay Inc. has adopted in its data centers we refer to these topologies as Topology 1, Topology 2, and Topology 3. In Topology 1, fault tolerance is achieved through two utility feeders owned by the same distribution company, with a backup uninterruptible power supply (UPS) that rides through any temporary glitches in the utility feeders. In Topology 2, fault tolerance is achieved with the same feeder arrangement and a backup system with a diesel genset and a UPS. Topology 3 uses loom Energy fuel cells as the primary supply to the IT loads, with two utility feeders providing the backup. We perform a comprehensive, quantitative comparison of key performance metrics, including reliability the probability that the system is able to continuously serve power to the IT load and availability the fraction of the system lifetime during which it is able to serve power to the IT load. To this end, we construct Markov models for each topology. These models incorporate only the failure and repair behavior of the components that have the dominant impact on the key performance metrics. The construction of the Markov models requires the identification of every configuration that results from each possible sequence of component failures, and for each of these configurations, the determination whether or not the IT load is served. The Markov analysis also allows the evaluation of the expected annual CO 2 emissions, and overall system efficiency. Our analysis uses data furnished by eay Inc., loom Energy, and Rocky Mountain Power. The values provided represent the most up-to-date values at the time our work was conducted. Our findings indicate that Topology 3 outperforms the other two topologies in terms of the reliability and the availability metrics. The reliability of Topology 3 is 0.998 a two-nine value represents a measurable improvement over that of Topology 2 at 0.98 a one-nine value. In other words, Topology 2 is on the order of ten times more likely to fail to serve the IT load than Topology 3. The least reliable topology is Topology 1, with a value of 0.90. The availability of Topology 3 is 0.999990, whereas that of Topology 2 is 0.99997. These values represent a significant improvement over the availability of Topology 1 at 0.998. These results imply that, over a year, on average, Topology 3 may be unable to serve the IT load a fraction of time of less than 6 minutes out of 8760 hours. The corresponding values for Topology 2 and Topology 1 are 15 and 599 minutes out of 8760 hours, respectively. In terms of the CO 2 footprint of each topology, our studies indicate that Topology 3 produces the least emissions. Indeed, the adoption of Topology 3 reduces the expected annual CO 2 emissions by 49 % from those under either of the other two topologies. This result makes sense since the emissions rate for the loom Energy fuel cell is about half that associated with the utility feeder supply. There is no difference in terms of the expected CO 2 emissions per year between Topology 1 or Topology 2. This result follows from the fact that Topology 2 relies on the diesel genset to serve the IT load whenever both utility feeders fail, and the emissions for the genset is approximately equal to that of the power supplied via the utility feeders. In terms of the overall efficiency, Topology 3 s efficiency at 52.5 % compares favorably with the 42.6 % of Topology 1 or Topology 2. This improvement with Topology 3 represents an increase of nearly 23 % with respect to the lower efficiency of the other two topologies. s such, the fuel consumption of the loom cell in Topology 3 is nearly 19 % below that of Topology 1 or Topology 2. The advantages with the fuel cell supply are expected to continue for several years, even as the penetration of wind and solar resources into the grid continues to deepen. The use of on-site fuel cells offers a significant energy efficiency benefit over that of the utility power-based Topology 1 or Topology 2.

nalysis of Design lternatives for Supplying Electric Power to Mission-Critical Loads in Data Centers: Reliability, Environmental, and Efficiency spects 1 lejandro D. Domínguez-García, George Gross, and Philip T. Krein Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign Urbana, IL 61801, US E-mail: {aledan, gross, krein}@illinois.edu bstract In this report, we analyze the reliability, carbon footprint, and overall efficiency of three different power supply topologies utilized by eay Inc. to serve mission-critical loads in data centers. In order to accomplish their functionality, the first and second topologies rely on two utility feeders as the main source for power, with conventional backup mechanisms an uninterruptible power supply (UPS) in the case of the first topology, and both a UPS and a bank of diesel generators in the case of the second topology. The third topology relies on loom Energy fuel cells as the main source of power for the mission-critical loads, with two utility feeders as backup. From a fault tolerance point of view, the first topology can accommodate, at most, one long-term failure in one of the two utility feeders. The second and the third topologies can accommodate up to two long-term failures. We perform a quantitative evaluation and comparison of the performance of the three topologies. To this end, we construct appropriate Markov models that allow us to compute all performance metrics of interest, including reliability, availability, and the expected number of hours per year that each topology is able to provide power to the mission-critical loads. In addition, we provide the expected values for the annual CO 2 emissions and the overall efficiency of each of the three topologies. The authors would like to thank Mike Lewis and James Monahan from eay Inc., and Jamie Day and Peter Gross from loom Energy for providing us with fruitful discussions and data, and helping us to clarify the numerous questions that we had while conducting the work reported in this document. ny opinions, findings, and conclusions or recommendations expressed here are those of the authors and do not necessarily reflect the views of the eay Inc. or loom Energy.

2 I. INTRODUCTION Reliable electric power supply is critically important in many applications, ranging from safety- and mission-critical systems for aerospace, aircraft, and automotive systems to information technology (IT) and communication systems. Typically, in any engineered system, the assurance of a high-level of reliability is achieved through the explicit incorporation of fault tolerant mechanisms into the system design. Fault tolerance is the ability of a system to adapt and compensate, in a systematic way, to random component faults, while providing complete or partial functionality [1]. The three basic features of a fault-tolerant system design are: (i) component redundancy; (ii) fault detection and isolation; and (iii) reconfiguration [2]. The reconfiguration feature ensures that, once a fault has been detected and isolated, a redundant component substitutes for a faulty one. In this report, we analyze the reliability of three fault-tolerant power supply topologies for mission-critical IT loads that eay Inc. has adopted in its data centers; we refer to these topologies as Topology 1, Topology 2, and Topology 3. In addition, we provide an assessment of the carbon footprint and the overall efficiency of each of these three power supply topologies. In Topology 1, fault tolerance is achieved through two utility feeders by the same utility company, with a backup uninterruptible power supply (UPS) that rides through temporary glitches in the utility feeders. In Topology 2, fault tolerance is achieved via two utility feeders owned by the same utility company, and a backup system comprising a UPS and a bank of diesel generators the Genset. Topology 3 relies on loom Energy fuel cells as the primary source of power to the IT loads, with two utility feeders owned by the same utility company as the backup. From a fault tolerance point of view, Topology 1 is one-fault tolerant, i.e., it can withstand, at most, one long-term failure. Topology 2 and Topology 3 are two-fault tolerant, i.e., each can withstand, at most, two long-term failures. We perform a comprehensive, quantitative comparison of the key performance metrics reliability, availability, and expected number of available hours per year for the three topologies. For this purpose, we construct appropriate Markov models for each topology; these models only incorporate the failure/repair behavior of the components that have the highest impact on the key performance metrics. Our analysis indicates the superiority of Topology 3 in terms of its reliability, i.e., the probability that the power supply system continuously delivers its functionality over an intended period of time [3]. Indeed, Topology 3 outperforms Topology 2 by one order of magnitude in terms of the reliability metric. useful by-product of the Markov model analysis is the computation of expected values of the annual emissions and efficiency metrics for each topology. For both these metrics, our analysis indicates that Topology 3 outperforms the other two topologies. However, it is important to note that the results yielded by the Markov models depend intensely on model parameters, i.e, the failure and repair rates of each component. The numerical values used in our analysis were provided by eay and loom Energy and represent their best estimates at the time our work was conducted [4]. Given the uncertainty associated with the values that these parameters may take, in future work we plan to explore the sensitivity of each performance metric to changes in the parameter values.

3 Utility Utility Utility Feeder a Feeder b Feeder a Feeder b Feeder a Feeder b Switch 1 Switch 1 Switch 1 us 1a us 1b us 1a us 1b us 1 Switch 2 UPS Genset Switch 2 Switch 3 UPS P f = 1200 kw Fuel cell Switch 2 IT load Supply 1: P s1 = 480 kw Supply 2: P s2 = 480 kw (a) Topology 1: power supplied through two parallel feeders. IT load Supply 1: P s1 = 480 kw Supply 2: P s2 = 480 kw (b) Topology 2: power supplied through two parallel feeders plus a cold-standby UPS/Genset. IT load Supply 1: P s1 = 480 kw Supply 2: P s2 = 480 kw (c) Topology 3: power supplied through onsite generation via Fuel cells plus two parallel feeders. Fig. 1. Data center mission-critical load power supply topologies. The remainder of this report is organized as follows. In Section II, we provide a qualitative description of the functionality of each of the three topologies. In Section III, we discuss the Markov models developed for evaluating the reliability, availability, and other performance metrics of interest of the three topologies a quantitative comparative analysis of the topologies is performed in terms of these performance metrics. In Section IV, we discuss the computation of the annual environmental impacts of each of the topologies. In Section V, we provide the quantitative comparison of the efficiency of each topology. Section VI summarizes the findings of our analyses. ppendix provides an overview on the modeling framework used in this work. ppendix lists the parameter values used in our analyses. II. DESCRIPTION OF POWER SUPPLY DESIGN LTERNTIVES This section provides an overview of three data center mission-critical load power supply topologies adopted by eay Inc. While many details of the actual physical elements in the topologies circuit breakers, switches, fuses, and transformers are omitted, the descriptions provided here capture the main functionality and salient features of each topology, and are adequate for use in our analysis.. Topology 1: Two Parallel Independent Utility Feeders The connectivity between sources and the mission-critical load for Topology 1 is shown in the simplified block diagram of Fig. 1(a). The power to the mission-critical load, referred to as IT load, can be supplied by two different sources, Supply 1 and Supply 2, both of which are enabled by a direct connection to us 1a and us 1b, respectively. The maximum power rating of the IT load is P l =960kW. Under normal operating conditions, half of the power, P s1 =480kW, is provided by Supply 1, while the other half, P s2 =480kW, is provided by Supply 2.

4 In the event that either Supply 1 or Supply 2 is unavailable due to some failure, the other supply will provide all 960 kw to the IT load. The power that flows into Supply 1 and Supply 2 via us 1a and us 1b, respectively, is provided in its entirety by two utility feeders, referred to as Feeder a and Feeder b, that belong to the same utility company. Under normal operating conditions, Feeder a supplies us 1a, which, in turn, serves power to Supply 1. Similarly, Feeder b serves power to Supply 2 via us 1b. If either Feeder a or Feeder b fails, then Switch 1 closes and both us 1a and us 1b are connected to the feeder that is still operational. The UPS helps ride through power glitches and feeder transfers, and if it fails, it is bypassed due to the action of Switch 2.. Topology 2: Two Parallel Dependent Utility Feeders plus a Cold-Standby Ups/Genset Figure 1(b) shows a simplified diagram for Topology 2, capturing the flow paths for the electric power sources to the IT load. The supply of power to the IT load is identical to that in Topology 1, with two different sources, Supply 1 and Supply 2. lso, as in Topology 1, both supplies are provided by a direct connection of Supply 1 and Supply 2 to us 1a and us 1b, respectively. Similar to Topology 1, under normal operating conditions, the total power to be delivered to the load, P l =960kW, is equally allocated to Supply 1 and Supply 2, with P s1 = P s2 =480kW. In the event that either of the supplies is unavailable, the total power is provided by the other available supply. Similarly to Topology 1, the power that flows into Supply 1 and Supply 2 via us 1a and us 1b, respectively, is provided entirely by two utility feeders, referred to as Feeder a and Feeder b, that belong to the same utility company. Under normal operating conditions, and similar to Topology 1, Feeder a and Feeder b supply power to us 1a and us 1b, respectively, which in turn supply power to Supply 1 and Supply 2, respectively. If either Feeder a or Feeder b fails, then Switch 1 closes and all power is supplied by the available feeder. In the event that a simultaneous failure of both feeders occurs, there is an uninterruptible power supply (UPS), and a diesel Genset for backup. The Genset is on cold standby, i.e., it is only brought online due the action of Switch 2 after both Feeder a and Feeder b become unavailable. lso, as in Topology 1, whenever the UPS fails, it is bypassed by the action of Switch 3. C. Topology 3: On-Site Generation via Fuel Cells plus two Independent Utility Feeders on Cold Standby Figure 1(c) shows a simplified diagram for Topology 3 that captures the connectivity between the power sources and the IT load. s shown in the lower part of the figure, the power to the IT load can be supplied by two different sources: (i) a Fuel cell, and (ii) a direct connection to us 1. The maximum power output of the Fuel cell is P f =1200kW, and it operates at that level under normal conditions. We refer to the Fuel cell supply as Supply 1, and to the direct connection to the us 1 supply as Supply 2. s shown in Fig 1(c), power injected into us 1 can be supplied by the Fuel cell and by two utility feeders, referred to as Feeder a and Feeder b, that belong to the same utility company. Under normal operating conditions, Feeder a is engaged and can inject or absorb power to or from us 1. Feeder b is engaged only if an outage in Feeder a occurs Switch 1 performs the transfer process. Note that the

5 Fuel cell can be bypassed through the action of Switch 2. Such a transfer occurs whenever the Fuel cell is not available due to either an internal failure or a failure in the gas supply. In such cases, the power to the IT load is provided by the utility supply via us 1 through the two utility feeders. The maximum power rating of the IT load is P l =960kW, and under normal operating conditions, half of this load, P s1 =480kW, is provided by Supply 1, and the other half, P s2 =480kW, is provided by Supply 2. In the event that either Supply 1 or Supply 2 is unavailable, the available power supply provides 960 kw. The Fuel cell can provide power to the IT load through Supply 1, and injects any excess power into us 1. Then, since the Fuel cell output is 1200 kw, and 480 kw is supplied through Supply 1, under normal operating conditions, the power provided to the IT load through Supply 2 (480 kw) is also provided by the Fuel cell, with the excess power (1200 960 = 240 kw) injected into the utility grid, or used to provide power to auxiliary loads not depicted in Fig. 1(c). In the unlikely event that both Feeder a and Feeder b are unavailable, the Fuel cell can supply all the power to the IT load by itself. III. RELIILITY/PERFORMNCE SSESSMENT In this section, we discuss the Markov models that we used to evaluate the reliability and performance metrics of the three topologies. The construction of these models relies on the description of the functionality of each of the three topologies provided in Section II; i.e., the models do not take into account the failure and repair behavior of the ancillary equipment such as, transformers, cables, circuit breakers, switches, fuses, and other protective devices. s such we use the same level of detail in the representation of the three topologies; thus, the analysis presented here is suitable for their comparison on a consistent basis. ppendix provides a brief summary of the Markov modeling framework used in this work; for a more detailed discussion on this topic, the reader is referred to [3], [5].. Topology 1: Two Parallel Independent Utility Feeders In the context of reliability analysis, Topology 1 is modeled as a a parallel system comprising the two utility feeders. However, these feeders are not independent as there are events, e.g., a wide-area blackout caused by a natural disaster, that may cause both feeders to fail at the same time. In order to build the reliability model for this topology, we first need to list all the configurations that arise from each possible sequence of failure events. For each configuration, we need to evaluate whether or not the system can maintain its functionality, which in this case means the continuation to provide power to the IT load. For this topology, there are four disjoint failure events that we need to consider when listing all possible failure sequences. These are: (i) Feeder a fails to supply power event, (ii) Feeder b fails to supply power event, (iii) both Feeder a and b fail to supply power at the same time event, and (iv) UPS fails to supply power event U. These events occur at rates,,, and U (see Table XII in ppendix for the numerical values used in the analysis). Once any one of these events occurs, we assume that a repair action may be triggered. The rates at which the

6 TLE I TOPOLOGY 1: FILURE EVENT DESCRIPTION. Event Description Failure rate Repair rate Event causing only Feeder a to fail to supply power µ Event causing only Feeder b to fail to supply power µ Event causing both Feeder a and b to fail to supply power µ U Event causing UPS to fail to supply power U µ U TLE II TOPOLOGY 1MRKOV RELIILITY MODEL: FILURE SEQUENCES, OUTCOMES, ND PROILITIES T t m = 8760 H. Configuration Failure sequence Power delivered to the load [kw] Probability at time t m = 8760 h 1 {;} 960 1 (t m )=0.901655791759185 2 {} 960 2 (t m )=0.000821562180308 3 {} 960 3 (t m )=0.000821562180308 4 {U} 960 4 (t m )=0.000020609275240 5 {} 0 5 (t m )=0.094780113755391 6 {! } 0 6 (t m )=0.000862779832013 7 {! } 0 7 (t m )=0.000086277983201 8 {! } 0 8 (t m )=0.000862779832013 9 {! } 0 9 (t m )=0.000086277983201 10 {U! } 960 10 (t m )=0.000000018778564 11 {U! } 960 11 (t m )=0.000000018778564 12 {U! } 0 12 (t m )=0.000002164318270 13 {U!! } 0 13 (t m )=0.000000019701692 14 {U!! } 0 14 (t m )=0.000000001970169 15 {U!! } 0 15 (t m )=0.000000019701692 16 {U!! } 0 16 (t m )=0.000000001970169 repair is completed after occurrence of events,,, and U are µ, µ, µ, and µ U, respectively (see Table XII in ppendix for the numerical values used in the analysis). key step in the analysis is to list all possible sequences of failure events, for which we use the following notation. We denote by {;} the event that no failures occurred. Sequences of failures involving single events,,, and U, are denoted by {}, {}, {}, and {U}. The sequence of failures that corresponds to event occurring first, followed by event, is denoted by {! }. Following this convention, the second column of Table II lists all possible failure sequences involving events,,, and U, while the third column lists the corresponding amount of power delivered to the IT load. y combining the information in Table I and Table II, we build a Markov model that allows us to evaluate the reliability of the topology (see, e.g., [3], [5]). The state-transition diagram for the resulting Markov reliability model is provided in Fig. 2, with the states corresponding to the configurations listed in Table II, and the transition rates in and out of each state indicated. y solving the Markov reliability model, we can compute the probability of occurrence of every sequence of failures at any given time instant. For example, the fourth column of Table II lists the probability of occurrence of each of the individual sequences of failures after t m =8760h have elapsed. In turn, these probabilities can be used to compute an estimate of the system reliability, which is defined as

7 2 {} 6 {! } 7 {! } µ 1 µ 3 8 {! } {;} {U} {} 4 µ µ 9 {! } 10 {U! } 13 {U!! } 14 {U!! } 5 {} 11 {U! } 12 {U! } 15 {U!! } 16 {U!! } Fig. 2. Topology 1: Markov reliability model state-transition diagram. Green states indicate sequences of failures that result in no loss of power delivery to the load. Red states indicate sequences of failures that result in loss of power delivery to the load. Definition 1 ([3]): Reliability is the probability that the system continuously delivers its functionality, i.e., the system is able to continuously supply the power requested by the IT load, for an intended period of time, referred to as the mission time. Let U denote the set of states of the Markov model in Fig. 2, the elements of which correspond to sequences of failures after which the system is still able to deliver power to the IT load, i.e., U = {1, 2, 3, 4, 10, 11}, (all other states correspond to sequences that result in failure). Then, by using Definition 1, for a mission time t m, the reliability of Topology 1 is given by R(t m )= X i2u i (t m )= 1 (t m )+ 2 (t m )+ 3 (t m )+ 4 (t m )+ 10 (t m )+ 11 (t m ). (1) With a one year mission time, i.e., t m =8760h; the values in the fourth column of Table II yield R(t m )=0.90. (2)

8 In addition to reliability, another performance metric of interest is availability, which is the fraction of the system lifetime that the system is delivering its functionality, i.e., providing power to the IT load. vailability can be obtained from the steady-state distribution of a Markov model derived by adding slight modifications to the one depicted in Fig. 2. Specifically, it is necessary to add the transitions out of all the states associated with configurations in which the system is no longer able to provide power; these transitions are associated with repair actions (see, e.g., [3], [5]). For example, for State 6, which corresponds to the failure sequence {! }, the transition that needs to be added would restore the system to operation in State 2 at the repair rate µ. The state-transition diagram of the modified Markov model, referred to as the Markov availability model, is displayed in Fig 3, with the values of the long-term probabilities listed in Table III. Let Q denote the set of states of the Markov availability model in Fig. 3 corresponding to sequences of failures after which the system is still able to deliver power to the IT load. Then, the availability of Topology 1 is given by: = X i2q i = 1 + 2 + 3 + 4 + 10 + 11 =0.998. (3) In addition, the expected number of hours within a year that the system delivers its functionality, which we denote by T, can be computed from as simply T =8760 =8750.01 h. TLE III TOPOLOGY 1 MRKOV VILILITY MODEL: FILURE SEQUENCES ND LONG-TERM PROILITIES Configuration Failure sequence Long-term probability 1 {;} 1 =0.997018249600004 2 {} 2 =0.000909280643635 3 {} 3 =0.000909280643635 4 {U} 4 =0.000022788988562 5 {} 5 =0.001136600804544 6 {! } 6 =0.000000829263947 7 {! } 7 =0.000001036579934 8 {! } 8 =0.000000829263947 9 {! } 9 =0.000001036579934 10 {U! } 10 =0.000000020783558 11 {U! } 11 =0.000000020783558 12 {U! } 12 =0.000000025979447 13 {U!! } 13 =0.000000000018955 14 {U!! } 14 =0.000000000023693 15 {U!! } 15 =0.000000000018955 16 {U!! } 16 =0.000000000023693

9 2 {} µ µ 6 {! } 7 {! } {;} µ µ 1 µ µ 3 {} 4 µ 8 {! } 9 {! } 10 {U} {U! } µ µ µ µ 13 {U!! } 14 {U!! } µ 5 {} 11 {U! } 12 {U! } µ µ 15 {U!! } 16 {U!! } Fig. 3. Topology 1: Markov availability model state-transition diagram. Green states indicate sequences of failures that result in no loss of power delivery to the load. Red states indicate sequences of failures that result in loss of power delivery to the load.. Topology 2: Two Parallel Dependent Utility Feeders plus a Cold-Standby Ups/Genset From a reliability point of view, Topology 2 can be modeled as a parallel system comprising two utility feeders, Feeder a and Feeder b, and a cold-standby backup comprising the UPS and the Genset. Upon failure of both Feeder a and Feeder b, the Genset is engaged by closing Switch 2; however, it is necessary that the UPS is available during this transfer process. In terms of their reliability, the two utility feeders are not independent since they belong to the same utility company; thus a blackout affecting this utility will affect both feeders. In this topology, there are five disjoint failure events that we need to consider. These are: (i) Feeder a fails to supply power event, (ii) Feeder b fails to supply power event, (iii) both Feeder a and Feeder b fail to supply power at the same time event, (iv) UPS fails to supply power event U, and (v) Genset fails to supply power event G. s displayed in Table IV, we use the same notation adopted for Topology 1 to denote the rates at which these failure events occur, and the corresponding repair rates (see Table XIII in ppendix for the numerical values used in the analysis).

10 TLE IV TOPOLOGY 2: FILURE EVENT DESCRIPTION. Event Description Failure rate Repair rate Event causing only Feeder a to fail to supply power µ Event causing only Feeder b to fail to supply power µ Event causing Feeder a and b to fail to supply power µ U Event causing UPS to fail to supply power U µ U G Event causing Genset to fail to supply power G µ G The next step in the analysis is to list all possible sequences of failure events and evaluate the outcome, i.e., whether or not the IT load is being supplied. The outcome of some failure sequences will depend on whether or not the Genset is successfully brought online. To this end, denote by S the event that describes success in bringing the Genset online, and denote by S the event that describes failure in bringing the Genset online. For example, consider the occurrence of event, i.e., both Feeder a and Feeder b fail at the same time and the Genset is successfully brought online; this sequence of events, denoted by { ^ S}, results in power being supplied to the IT load. On the other hand, if the Genset is not successfully brought online after event occurs (we denote this sequence of events by { ^ S}), then the system will fail to provide power. To illustrate further, {! ^ S! G} denotes the following sequence of events: Feeder a fails first; then, Feeder b fails and the Genset is successfully brought online; and finally the Genset fails. Following this convention, the second column of Table V lists all possible sequence of failures, while the third column lists the outcome of each sequence. To build the Markov reliability model, we need to define the probability 1 associated with event S, which we denote by c S. It follows that the probability associated with event S is 1 c S. y combining the information in Table IV and Table V, we can build the Markov reliability model state-transition diagram shown in Fig. 4. y using the failure and repair values in Table XIII in ppendix, and assuming that c S =0.99, we can compute the probabilities of each of the states for t m =8760h; these probabilities are listed in the fourth column of Table V. The next step is to use the results of the Markov reliability model to compute the reliability. s in the analysis of Topology 1, we can modify the transition diagram in Fig. 4 to generate a Markov availability model. The long-term occupational probabilities for this model are listed in the fifth column of Table V. The reliability, availability, and expected number of available hours per year estimates are listed in Table VI. C. Topology 3: On-Site Generation via Fuel Cells plus two Independent Utility Feeders in Cold Standby From a reliability point of view, Topology 3 can be be modeled as a parallel system comprising Feeder a and the Fuel cell, and a cold-standby backup system comprising Feeder b that is engaged when Feeder a fails. s in Topology 1 and Topology 2, the two utility feeders belong to the same utility company; however, unlike Topology 1 and Topology 2, only one of them is engaged at a time. 1 In the context of fault-tolerant systems, this probability is referred to as failure coverage (see, e.g., [6], [7]), and in general captures the ability of a fault-tolerant system to recover from a component failure and stay operational.

11 {;} µ U µ µ 1 {} {} c S 2 µ µ (1 c S ) 3 4 { ^ S} 5 c S µ µ c S µ c S 7 {! ^ S} 9 {! ^ S} 10 { ^ S} { ^ S! G} U {U} µ U µ U 6 8 (1 c S ) {! ^ S} (1 c S ) U c s (1 c S ) {! ^ S} 12 13 (1 c s ) {! ^ S} U G µ µ 11 {! U} {! ^ S} 14 {! ^ S} 15 {! ^ S} 16 {! U} 17 18 {U! } 19 {U! } 20 {U! } 22 {! U! } 23 {! U! } 24 {! U! } 25 {! U! } 26 {U!! } 27 28 {U!! } 29 G G 21 G G {U!! } {U!! } {! ^ S! G}_ {! ^ S! G}_ {! ^ S! G}_ {! ^ S! G} Fig. 4. Topology 2: Markov reliability model state-transition diagram. Green states indicate sequences of failures that result in no loss of power delivery to the load. Red states indicate sequences of failures that result in loss of power delivery to the load.

12 TLE V TOPOLOGY 2: FILURE SEQUENCES, SSOCITED OUTCOME, MRKOV RELIILITY MODEL PROILITIES T t m = 8760 H, ND MRKOV VILILITY MODEL LONG-TERM PROILITIES. Configuration Failure sequence Power delivered Markov reliability model Markov availability model to the load [kw] probabilities at t m = 8760 h long-term probabilities 1 {;} 960 0.980087888618362 0.997002133659019 2 {} 960 0.000893831472938 0.000909265945897 3 {} 960 0.000893831472938 0.000909265945897 4 { ^ S} 960 0.000921925996283 0.001125216608048 5 { ^ S} 0 0.000987370330719 0.000011365824324 6 {U} 960 0.000022400283345 0.000022788620198 7 {! ^ S} 960 0.000000794326027 0.000000815140224 8 {! ^ S} 0 0.000008996426478 0.000000008292505 9 {! ^ S} 960 0.000000079432603 0.000000087913617 10 {! ^ S} 0 0.000000899642648 0.000000010365632 11 {! U} 960 0.000000020410284 0.000000020783222 12 {! ^ S} 960 0.000000794326027 0.000000815140224 13 {! ^ S} 0 0.000008996426478 0.000000008292505 14 {! ^ S} 960 0.000000079432603 0.000000087913617 15 {! ^ S} 0 0.000000899642648 0.000000010365632 16 {! U} 960 0.000000020410284 0.000000020783222 17 { ^ S! G} 0 0.016137956124146 0.000018003465729 18 {U! } 960 0.000000020408910 0.000000020783222 19 {U! } 960 0.000000020408910 0.000000020783222 20 {U! } 0 0.000002254590106 0.000000025979027 21 {! ^ S! G}_ 0 0.000030829514482 0.000000007224431 {! ^ S! G}_ {! ^ S! G}_ {! ^ S! G} 22 {! U! } 0 0.000000020524057 0.000000000018954 23 {! U! } 0 0.000000002052406 0.000000000023693 24 {! U! } 0 0.000000020524057 0.000000000018954 25 {! U! } 0 0.000000002052406 0.000000000023693 26 {U!! } 0 0.000000020522661 0.000000000018954 27 {U!! } 0 0.000000020522661 0.000000000018954 28 {U!! } 0 0.000000002052266 0.000000000023693 29 {U!! } 0 0.000000002052266 0.000000000023693 TLE VI TOPOLOGY 2: PERFORMNCE METRICS. Reliability vailability Expected number of available hours per year 0.98 0.99997 8759.74 h

13 TLE VII TOPOLOGY 3: FILURE EVENT DESCRIPTION. Event Description Failure rate Repair rate Event causing only Feeder a to fail to supply power µ Event causing only Feeder b to fail to supply power µ Event causing Feeder a and b to fail to supply power µ F Event causing Fuel cell to fail to supply power F µ F F Event causing Feeder a and b, and Fuel cell to fail to supply power F µ F TLE VIII TOPOLOGY 3: FILURE SEQUENCES, SSOCITED OUTCOME, MRKOV RELIILITY MODEL PROILITIES T t m = 8760 H, ND MRKOV VILILITY MODEL LONG-TERM PROILITIES. Configuration Failure sequence Power delivered Markov reliability model Markov availability model to the load [kw] probabilities at t m = 8760 h long-term probabilities 1 {;} 960 0.996022874571926 0.997017054404012 2 {} 960 0.000908372846032 0.000909279553616 3 {} 960 0.000908372846032 0.000909279553616 4 {F } 960 0.000012579620843 0.000012593899635 5 {} 960 0.001135359646679 0.001136599442021 6 {F } 0 0.000995185027564 0.000011365994420 7 {! } 960 0.000000828429823 0.000000829262953 8 {! F } 960 0.000000011458524 0.000000011485636 9 {! F } 0 0.000000906777746 0.000000010365787 10 {! } 960 0.000000828429823 0.000000829262953 11 {! F } 960 0.000000011458524 0.000000011485636 12 {! F } 0 0.000000906777746 0.000000010365787 13 {F! } 960 0.000000011461127 0.000000011485636 14 {F! } 0 0.000001255179936 0.000000014357046 15 {! F } 0 0.000010355030324 0.000000014357046 16 {!! F } 0 0.000000007628983 0.000000000010475 17 {! F! } 0 0.000000011422733 0.000000011485636 18 {!! F } 0 0.000000007628983 0.000000000010475 19 {! F! } 0 0.000000011422733 0.000000000010475 20 {F!! } 0 0.000000011425327 0.000000000010475 21 {! } 960 0.000001035447980 0.000001036578691 22 {! } 960 0.000001035447980 0.000001036578691 23 {!! F } 0 0.000000009435041 0.000000000013094 24 {!! F } 0 0.000000009435041 0.000000000013094 25 {F!! } 0 0.000000001142533 0.000000000013094 The disjoint events to consider in the analysis are: (i) failure of Feeder a event, (ii) failure of Feeder b event, (iii) Feeder a and Feeder b becoming simultaneously unavailable event, (iv) failure of the Fuel cell event F, and (v) Feeder a, Feeder b, and Fuel cell becoming simultaneously unavailable event F. s displayed in Table VII, we use a similar notation to the one in earlier analyses to denote rates at which these failure events occur and corresponding repair rates (see Table XIV in ppendix for the numerical values used in the analysis).

14 {} 2 µ µ F F 7 {! } 8 F 16 {!! F } 17 µ {! F } {! F! } 21 {! } F 9 {! F } F 23 {!! F } µ {;} µ µ 1 3 µ µ F F {} µ µ F F 10 {! } 11 {! F } F 18 {!! F } 19 {! F! } µ 22 F 12 {! F } {! } F 4 {F } µ F 24 {!! F } F µ 5 {} 6 {F } F 13 {F! } 14 {F! } 15 20 {F!! } 25 {F!! } {! F } Fig. 5. Topology 3: Markov reliability model state-transition diagram. Green states indicate sequences of failures that result in no loss of power delivery to the load. Red states indicate sequences of failures that result in loss of power delivery to the load. To build the Markov reliability model, we list all possible sequences of failure events and evaluate the outcome; see the second column of Table VIII for the list, and the third column for the outcome. Then, by combining the information in Table VII and Table VIII, we can build the Markov reliability model state-transition diagram shown in Fig. 5. y using the failure and repair values in Table XIV in ppendix, we can evaluate the model and compute the probability of occurrence of each of the individual sequences of failures after t m =8760h have elapsed (see the fourth column of Table VIII for the values that these probabilities take). We can modify the transition diagram in Fig. 5 to generate a Markov availability model and compute the long-term probabilities for this model (see the fifth column of Table VIII). The same three performance metrics are computed and summarized in Table IX.

15 TLE IX TOPOLOGY 3: PERFORMNCE METRICS. Reliability vailability Expected number of available hours per year 0.998 0.999990 8759.90 h TLE X TOPOLOGY COMPRISON. Topopolgy 1 Topology 2 Topology 3 Reliability 0.90 0.98 0.998 vailability 0.998 0.99997 0.999990 D. Comparative nalysis We summarize the metrics evaluated for the three topologies in Table X. We find that Topology 3 outperforms both Topology 1 and Topology 2 in each metric. The reliability of Topology 3 is 0.998, i.e., two nines. The reliability of Topology 2 is 0.98, i.e., one nine. The least reliable topology is Topology 1, with a reliability of 0.90, i.e., one nine. From a redundancy point of view, Topology 1 is essentially a dual-redundant system with Feeder a and Feeder b sharing the power supply to the IT load, and each one able to provide full power to the IT load in case of failure of the other feeder. Topology 2 is a dual-redundant system with Feeder a and Feeder b in a similar arrangement as in Topology 1. Topology 2 has the additional feature of a cold-standby Genset backup, engaged whenever both Feeder a and Feeder b fail. Topology 3 differs significantly from Topology 1 or Topology 2. Specifically, while in theory Feeder a and the Fuel cell constitutes a dual-redundant system as in the Topology 1 and 2, under normal operating conditions, the Fuel cell is capable of providing all the power to the IT load. s such, Feeder a is essentially acting as a hot standby system for the Fuel cell. lso Feeder b acts as a cold standby system for Feeder a. From a fault tolerance point of view, Topology 1 is one fault tolerant, i.e., it can withstand, at most, one long-term failure. Topology 2 and Topology 3 are two fault tolerant, i.e., each can withstand, at most, two long-term failures. The level of fault tolerance partly explains why Topology 1 is the less reliable topology. However, while in Topology 2 the Genset provides one more level of fault tolerance, the likelihood of the Genset providing power to the IT load when both Feeder a and Feeder b fail is a function of both the Genset failure rate and the probability that the Genset is successfully brought online in case of need. Indeed, these two factors constitute the reason that explains the superiority of Topology 3 over Topology 2. While the failure rate for the feeders is the same in either topology, the failure rate for the Genset is three orders of magnitude higher than the failure rate of the Fuel cell. Moreover, unlike the probability of successfully bringing the Genset online, the probability of successfully engaging Feeder b in Topology 3 is taken to be one. This discussion provides a succinct summary for the gains in performance attainable with Topology 3 vis-a-vis Topology 1 and Topology 2.

16 IV. CRON FOOTPRINT SSESSMENT To assess the CO 2 footprint of the three topologies, we need (i) the long-term probabilities of each possible configurations resulting from a particular sequence of failures, and (ii) the power provided under each possible configuration. The long-term probabilities for each topology are given in the previous section. These probabilities are listed in Table III for Topology 1, Table V for Topology 2, and Table VIII for Topology 3. The emissions assessment requires the data for the emissions rates associated with each power source: Feeder a, Feeder b, Genset, and Fuel cell. We denote these emissions rates by e a, e b, e g, and e f, respectively, and list their values in Table XV in ppendix. The numbers used for Feeder a and Feeder b are for the year 2012 and were provided by Rocky Mountain Power, the utility that serves the eay Inc. data centers in Utah, for the year 2012. We interpret the probability of each possible operational configuration as the fraction of time that a configuration associated with a particular failure sequence occurs during a specified period of time t e, say a year. For instance, in Topology 3, the long-term probability associated with failure sequence {}, i.e., Feeder a fails first, is 2 =0.00091 as shown in Table VIII. Then, since in that configuration, the Fuel cell is operating at its maximum power output 2 P f =1200kW, the yearly contribution of the emissions associated with that particular configuration is E 2 (t e )= 2 t e P f e f =0.00091 8760 h 1200 kw 0.773 lbs/kwh =7.39 10 3 lbs. (4) In (4), we consider only the emissions of the Fuel cell as this is the sole source of power for the IT load. However, there are other configurations in which more than one source provides power to the IT load. In Topology 1, when no failures occurred, i.e., the system is operating in Configuration 1 (see Table II), half of the power is supplied to the IT load by Feeder a, and the other half is provided by Feeder b. The yearly CO 2 emissions associated with that particular configuration of Topology 1 are E 1 (t e )= 1 t e (P s1 e a + P s2 e b ) =0.997 8760 h (480 kw 1.519 lbs/kwh +480 kw 1.519 lbs/kwh) =1.27 10 7 lbs. (5) Then, once all the E i (t e ) s are evaluated, for each topology, we can compute the expected CO 2 emissions per year using: E(t e )= X i E i (t e ), (6) with t e =8760h. 2 Recall from that description in Section II- that P l =960kW only are delivered to the IT load, while any excess power that the Fuel cell produces, i.e., 1200 960 = 240 kw, is injected into the utility grid through either Feeder a or Feeder b or used to provide power to data center auxiliary loads.

17 TLE XI EXPECTED YERLY CO 2 EMISSIONS Topology 1 Topology 2 Topology 3 Expected CO 2 emissions per year [lbs] 1.28 10 7 1.28 10 7 6.50 10 6 For each of the three topologies, Table XI lists the values of E(t e ). s one can see from the results in this table, Topology 3 outperforms the other topologies with an expected CO 2 emissions per year about half the expected CO 2 emissions per year of either of the other topologies. This result makes sense by observing that the emissions rate for the Fuel cell is half the emissions rate associated with the utility feeders of both Topology 1 and Topology 2. There is no difference in terms of the expected CO 2 emissions per year of Topology 1 or Topology 2. This result is clear from the fact that Topology 2 relies on the Genset to provide power whenever both Feeder a and Feeder b fail, and the emissions rate for the Genset is similar to that of the power generated and delivered via the utility feeders, with the Genset rarely functioning more than a few hours per year. V. OVERLL ENERGY EFFICIENCY SSESSMENT Electricity in Utah comes primarily from coal-fired generation plants, although, recently, there has been substantial growth in the implementation of natural gas and wind resources. Thermal energy efficiency for coal plants is typically about 33 %, but the actual generation mix in Utah has significant other resource penetration. We compute an equivalent fuel efficiency based on CO 2 production. Rocky Mountain Power, which serves the sites of the eay Inc. data centers, reported CO 2 emissions rate to be approximately 1.52 lbs/kwh [4]. The Energy Information dministration reports that in Utah, coalbased electricity production emits about 204 lbs of CO 2 per million TU [8]. Now, one million TU is the energy equivalent of about 293 kwh and so the coal-based generation at 100 % efficiency produces 0.696 lbs/kwh. The reported amount of 1.52 lbs/kwh implies that the average coal-equivalent efficiency is 0.696/1.52 = 45.8 %. However, the actual efficiency is below this value, in part because natural gas combustion involves lower CO 2 emissions than coal combustion. In our analysis, we use the 45.8 % coal-equivalent efficiency value as it offers the same basis for efficiency comparison that the CO 2 emission do. ny electrical energy from a generation source is delivered via the transmission and distribution networks, and possibly a user-side conversion device before it reaches the customer load where it is consumed. oth coal and natural gas have distribution costs and associated energy requirements. The comparison in this report is not intended as a wells to wheels computation. Rather, the efficiency and energy consumption are compared based on the point of entry of the electricity into the data center site. For the topologies depicted in Fig. 1, and some additional information on the components of the designs, we know that the electricity flow in Topology 1 and in Topology 2 passes from the point of entry into the data center site through two sets of transformers and the UPS before it is used to meet

18 the data center IT load. Distribution transformers in a [0.5, 1.5] MW rating range have efficiencies that approach 99 % [9], and we use this value in our analysis. The UPS efficiency is given to be 95 %. The efficiency from site entry to data center IT load is, therefore, 0.990 0.990 0.950 = 0.931, or 93.1 %. The coal equivalent efficiency is obtained by multiplication by the 0.458 value, to yield 0.426 or the 42.6 % overall energy efficiency from fuel to the data center IT load based on the energy contents of coal. On the other hand, for Topology 3, under normal operating conditions, power is supplied by the loom Energy fuel cell, whose output is 1200 kw. We need to determine the fuel cell nominal full-load efficiency, which we do using reverse engineering. The conversion from the natural gas thermal source to ac electricity is reported to be 53 % [4], which includes the efficiency of the associated inverter. The ac power feeds into the data center IT load after passing through a transformer. Overall efficiency from the natural gas supply at site entry to electricity to the data center IT load through the fuel cells, inverter, and the single transformer is computed to be 0.53 0.99 = 0.525, or 52.5 %. The 52.5 % overall efficiency in Topology 3 compares favorably with the overall 42.6 % efficiency of Topology 1 and Topology 2, and represents an increase of nearly 23 % with respect to the lower efficiency. s such, the fuel consumption of the fuel cell in Topology 3 is nearly 19 % below that of Topology 1 and Topology 2. The advantages with the fuel cell supply is expected to continue for several years, even as the implementation of wind and solar resources into the grid continues to expand. The use of on-site fuel cells offers a significant energy efficiency benefit compared to the utility power-based Topology 1 or Topology 2 supply. VI. CONCLUDING REMRKS In this report, we analyzed the reliability, carbon footprint, and overall efficiency of the three different power supply topologies adopted by eay Inc. to serve mission-critical loads in data centers. The first two topologies rely on utility feeders as the main source of power, while the third one relies on loom Energy fuel cells as the main power supply, with two utility feeders as backup. For each topology we constructed Markov models to describe their uncertain behavior due to component failures and repairs. These Markov models allowed us to compute performance metrics of interest, including reliability, availability, expected number of available hours per year, and expected CO 2 emissions per year. In turn, these key metrics allowed us to quantitatively compare the performance of the topologies from three perspectives. For each performance metric considered in the analysis, Topology 3, which relies on fuel cells as the main source of power, outperformed the other two. However, since the results depend on the model parameter values, future work needs to carefully explore the sensitivity of performance metrics with respect to parameter values.

19 PPENDIX MRKOV MODELS FOR RELIILITY ND PERFORMNCE EVLUTION Let X = {X(t), t 0} denote a Markov chain describing the stochastic behavior of a system subject to component failures and repairs. ssume that X takes values in a finite set S = {1, 2,..., n}, where 1 indexes the nominal, non-faulty system configuration, and 2,..., n index system configurations that arise due to sequences of component failures. Let i (t), t 0, be the probability that the system is in configuration i, and define the corresponding probability vector as (t) =[ 0 (t), 1 (t),..., n (t)]. The evolution of (t) is defined by the Chapman-Kolmogorov equations (t) = (t), (7) with 1 (0) = 1, j (0) = 0, j=2,..., n, and where is the generator of X (see, e.g., [11], [12]). To determine, the first step is to list all possible configurations associated with different component failure sequences. Transitions between states are governed by a combination of failure and repair rates. 1 2 {;} {} (a) Reliability model. µ 1 2 {;} {} (b) vailability model. Fig. 6. Single-component system Markov reliability and availability models. Example 1: Consider a system comprising a single component, which we denote by Component a, and denote by an event that causes Component a to fail. This system can only adopt two possible configurations, one in which Component a is operating, and one in which Component a has failed; thus S = {1, 2}. Figure 6(a) shows the state-transition diagram of the Markov reliability model associated with this system, where {;} denotes the sequence of component failures (in this case the empty set) that lead to configuration 1, and {} denotes the sequence of component failures, in this case only involving Component a, that lead to configuration 2. From Fig. 6(a), the generator for this Markov reliability model is given by " # = 0 0, (8) where is the failure rate for Component a. y solving (7) for 1 (0) = 1, and 2 (0) = 1, we obtain 1 (t) =e t, 2 (t) =1 e t, (9) and, for any mission time t m, the reliability of the system can be computed as R(t m )= 1 (t m )=e t m. (10)

20 Transitions back from state 2 in Fig. 6(a) represent a repair action. This will yield a Markov availability model for this system (Fig. 6(b) displays the corresponding state-transition diagram). Then, it follows from Fig. 6(a) that the generator of the Markov availability model is " # = µ µ, (11) where µ is the repairt rate for Component a. The stationary distribution of the Markov availability model, =[ 1, 2 ], can be obtained by solving =0with 1 + 2 =1, from which it follows that 1 = Finally, the availability of this system is µ µ +, 2 = = 1 = PPENDIX PRMETER VLUES USED IN THE NLYSES µ +. (12) µ µ +. (13) Tables XII XV display the parameter values used in our analyses. These numbers were provided by eay Inc. and loom Energy and represent their best estimates at the time the work was conducted [4]. TLE XII TOPOLOGY 1: FILURE ND REPIR RTES NUMERICL VLUES. Label Description Failure rate [h 1 ] Repair rate [h 1 ] Event causing only Feeder a to fail to supply power =1.14 10 4 µ =1.25 10 1 Event causing only Feeder b to fail to supply power =1.14 10 4 µ =1.25 10 1 Event causing both Feeder a and b to fail to supply power =1.14 10 5 µ = 10 2 U Event causing UPS to fail to supply power U =2.86 10 6 µ U =1.25 10 1 TLE XIII TOPOLOGY 2: FILURE ND REPIR RTES NUMERICL VLUES. Event Description Failure rate [h 1 ] Repair rate [h 1 ] Event causing only Feeder a to fail to supply power =1.14 10 4 µ =1.25 10 1 Event causing only Feeder b to fail to supply power =1.14 10 4 µ =1.25 10 1 Event causing Feeder a and b to fail to supply power =1.14 10 5 µ = 10 2 U Event causing UPS to fail to supply power U =2.86 10 6 µ U =1.25 10 1 G Event causing Genset to fail to supply power G =2 10 3 µ G =1.25 10 1

21 TLE XIV TOPOLOGY 3: FILURE ND REPIR RTES NUMERICL VLUES. Event Description Failure rate [h 1 ] Repair rate [h 1 ] Event causing only Feeder a to fail to supply power =1.14 10 4 µ =1.25 10 1 Event causing only Feeder b to fail to supply power =1.14 10 4 µ =1.25 10 1 Event causing Feeder a and b to fail to supply power =1.14 10 5 µ = 10 7 F Event causing Fuel cell to fail to supply power F =1.05 10 6 µ F =1.25 10 1 F Event causing Feeder a and b, and Fuel cell to fail to supply power F =1.14 10 7 µ F = 10 2 TLE XV POWER SOURCES EMISSIONS RTES [4]. Utility feeder Genset Fuel cell Emission rate [lbs/kwh] 1.519 1.700 0.773 REFERENCES [1] J. Laprie, Dependability: asic Concepts and Terminology. New York, NY: Springer-Verlag, 1991. [2]. vizienis, Toward systematic design of fault-tolerant systems, IEEE Computer, vol. 30, no. 4, pp. 51 58, pr 1997. [3] J. Endrenyi, Reliability Modeling in Electric Power Systems. Chichester, England: John Wiley & Sons, 1978. [4] eay Inc., private communication, ugust 2013. [5] M. Rausand and. Høyland, System Reliability Theory. Hoboken, NJ: Wiley Interscience, 2004. [6] W. ouricius, W. Carter, and P. Schneider, Reliability modeling techniques for self-repairing computer systems, in Proceedings of the 24 th National CM Conference. New York, NY: CM Press, 1969. [7] T. rnold, The concept of coverage and its effect on the reliability model a repairable system, IEEE Transactions on Computers, vol. C-22, no. 3, pp. 251 254, March 1973. [8]. D. Hong and E. R. Slatick, Carbon dioxide emission factors for coal, Quarterly Coal Report, U.S. Energy Information dministration, Report DOE/EI-0121, January-pril 1994. [9] S. Fassbinder, Efficiency and loss evaluation of large power transformers, Leonardo Energy, European Copper Institute, May 2013. [Online] http:// http://www.leonardo-energy.org. [10] Data sheets, SatCon PowerGate Plus PVS-500. [Online] http://www.civicsolar.com. [11] J. R. Norris, Markov Chains. New York, NY: Cambridge University Press, 1997. [12] G. Grimmett and D. Stirzaker, Probability and Random Processes. Oxford University Press, 1992.