Internet reliability with realistic peering

Transcription

1 Environment and Planning B: Planning and Design 2006, volume 33, pages 325 ^ 343 DOI: /b3187 Internet reliability with realistic peering Morton E O'Kelly, Hyun Kim Department of Geography, The Ohio State University, Columbus, OH , USA; okelly.1@osu.edu, kim.1567@osu.edu Changjoo Kim Department of Geography, Minnesota State University, 7 Armstrong Hall, Mankato, MN 56001, USA; changjoo.kim@mnsu.edu Received 28 July 2004; in revised form 3 June 2005 Abstract. Peering between network providers is modeled as a set of interconnections at hub nodes. The paper employs probabilistic methods to determine the likelihood of a path between nodes, under a variety of simulated disruptions or failures of parts of a network. Reliability is measured as the probability of withstanding failure of network components such as nodes or linkages. A reliability envelope is devised in order to define a range of impactsöwith the recognition that the damage to subsets of the network can vary from a relatively benign effect to a much more critical or lethal impact. Thus, we describe a range of scenarios from best to worst, within the same level of infrastructure loss. With the use of simulation data for the United States, results show the critical role of major hubs in sustaining communication among cities. In the paper we highlight especially reliable nodes (for example, Chicago and Atlanta) and those that are most susceptible (for example, Birmingham, Alabama, and Nashville, Tennessee). Failure in major hub cities such as Chicago, Dallas, and San Francisco would significantly influence adjacent areas and could cause severe degradation for the entire network with a loss of resiliency to below tolerable levels. An important finding is that various patterns in the reliability envelope can be influenced by factors such as network structures, hub structures, and geographical locations. 1 Introduction The Internet, a highly interconnected computer network system, comprises a number of Internet backbone providers (IBPs) (1) and Internet service providers (ISPs). The backbone providers mainly play a role of operating long-haul transmission trunk lines with high bandwidths (2) in order to carry Internet traffic at a national level. By connecting to these trunk lines, ISPs can provide Internet service to their customers, even if their infrastructure is present only at a regional level. The current Internet operates by exchanging traffic with the help of peering, a particular structure of mutual interconnections between multiple providers. These relationships are located at Internet exchanges or access points in some select cities. These city nodes may be called Internet hubs as they play a critical role as points for exchanging information flows originating from various IBPs. Internet hubs are important in that they enhance the ability of a network to transfer traffic in an efficient manner. At the same time, such hub nodes are vulnerable to an attack that might lead to total loss of their connecting function. It is important to keep the operational functionality of the Internet at a high level. The degradation of network communications caused by the attack at the World Trade (1) IBPs are often referred to as `global Internet service providers' because they mainly transit huge amounts of Internet traffic at a national level (Huston, 1999). (2) Bandwidth is commonly defined as the information-carrying capacity of a link as measured in bits per second for transferring traffic. Massive investment for the growth in backbone capacity is among the most prominent trends in Internet development (Huston, 1999; Malecki, 2002).

2 326 M E O'Kelly, H Kim, C Kim Center in New York highlights the issue of how vulnerable the Internet is to unexpected damage. The damage in New York City gave rise to short-term failures of statewide public online services for parts of the adjacent region. Further, it led to temporal network degradation for broader telecommunication services owing to the local loss of fiber optic lines, but at the same time the network exhibited remarkable resilience (see Grubesic et al, 2003; NYSDPS, 2002). In another example, a train crash in Baltimore's Howard Street tunnel in July 2001 caused the loss of fiber cables connecting two ISPs. Internet traffic in Washington ^ Baltimore became so congested that it had a significant impact on telecommunication quality on the East Coast, and this spread as far away as Seattle and Los Angeles. This incident also stressed the importance of peering arrangements with sufficient capacity on linkages among ISPs to make the Internet more resilient to disruption (Broadband Week 2001). The main purpose of this paper is to examine the vulnerability of the Internet in terms of pairs of major city nodes in the United States by taking into account peering arrangements in select hubs. In particular, in this paper we introduce the concept of reliability as a measurement to evaluate the vulnerability of the Internet as well as the geographical significance of selected network nodes. This paper is organized as follows. The following section explores issues relevant to an examination of the vulnerability of the Internet and to related analyses shown in previous research. Section 3 provides a brief explanation of reliability measurement as well as the characteristics of Internet hubs. The process of our model is demonstrated with the use of empirical analyses based on select IBPs in the United States. Concluding remarks are given in the final section. 2 Literature To our knowledge, the idea of a survivable network was initially addressed for various network configurations by Baran (1964). Since that time various approaches and measures have been proposed to either identify or assess network weaknesses (Ball et al, 1989; Coburn et al, 1994; Curley and Sha, 1982; Shake et al, 1999). In recent years the stability and tolerance of complex topologies such as scale-free and exponential networks with respect to random failures have been explored using probabilistic approaches. The results show a remarkable resiliency of large-size networks against random disruption, but also stress that current telecommunication systems could be vulnerable to failures at vital hubs (Albert and Barabäsi, 2002; Albert et al, 2000; Callaway et al, 2000; Yook et al, 2001). Work on the vulnerability of the Internet has also been found recently in geography. Geographers working on Internet backbones created a ranking for US cities in terms of measurements such as connectivity and graph theory (Malecki and Gorman, 2001; Moss and Townsend, 2000). The differences of city accessibilities and topological structures of the US commercial Internet were also investigated through the use of network analysis (O'Kelly and Grubesic, 2002; Wheeler and O'Kelly, 1999). Other work measures the growth of the Internet by comparing bandwidths of network links for cities or metropolitan areas (Malecki, 2002). The previous research mainly focuses on the potential availability for each commercial Internet network. The common characteristics in this work is that it assumes that all backbone networks of the Internet operate normally without any failures of network components such as links and nodes. Potential availabilities of a network are generally expressed in the form of discrete scores such as the number of paths, accessibility indices, and total volumes of capacities such as bandwidth. Therefore, the performance of commercial networks can be evaluated by comparing their relative values.

3 Internet reliability with realistic peering 327 More recent work addresses survivability as another approach for examining the vulnerability of the Internet. Grubesic et al (2003) address the question of how the vulnerability of the commercial Internet would change if some selected nodes were closed. Survivability, in their work, concerns the smallest amount of damage that results in the disconnection of a network, or reduces its performance to a level comparable with the levels of total failure. However, the all-or-nothing assumption, either 100% operation or total loss of nodal functionality, applied in Grubesic et al (2003) is unsatisfactory considering the fact that each city hub has a subnetwork in which interconnections among IBPs are created according to their peering agreements. Physical `disjoint analysis' as a vulnerability measure was performed to identify the location of critical infrastructure (Gorman et al, 2004). The simulation mainly examined impacts from the loss of original connections on fiber-optic networks. However, the previous measures associated with the vulnerabilities of the Internet need to be refined by taking into account peering arrangements in select hubs. As a network experiences an increasing number of interconnection failures at hubs, reliabilities will be presented probabilistically. In this paper we examine stochastic variations of reliabilities by considering peering arrangements among IBPs at hubs instead of an all-or-nothing assumption for nodes. 3 Network reliability, interconnection in hubs, and the reliability envelope 3.1 Network reliability Network reliability refers to the ability of a network to withstand the failure of components such as nodes or linkages (Colbourn, 1987). As this idea was originally conceived in order to evaluate the performance of a network when carrying out a desired operation, it is mainly applied with regard to transportation and electric circuits. The degree of network reliability is expressed in the form of probabilities for successful communications among specific pairs of nodes. Table 1 shows the general classification of reliability and corresponding terms depending on how many nodes are considered for measuring probabilistic connectedness. For example, all-terminal reliability is used to examine how well every node in a network is connected in terms of a spanning tree structure. Among the different measures of reliability, in this paper we stress the concept of origin ^ destination reliability because our concern is to compare previous work in terms of the individual level, such as city accessibilities, by measuring reliability for pairs of city nodes. Two simple cases, serial and parallel arrangements, are exemplified in figure 1 (over) to compute the reliability between two nodes A and B. Suppose that the operational probability of `p ˆ 0:9' is given for each linkage x 1 and x 2 as a parameter to compute reliability. An alternative interpretation for this parameter is that each linkage has capacity for 90% of traffic to flow without Table 1. Types of reliability measurements (sources: Colbourn, 1987, Shier, 1991). Type Definition Origin ± destination reliability The probability that there exists at least one origin to destination path for a pair of nodes in a network k-terminal reliability The probability that the network contains paths between (origin ± k destinations) each pair of the k nodes (2 4 k 4 n) All-terminal reliability The probability that the network contains at least a spanning tree for every pair of nodes Reachability The probability that there are paths from the `origin' to all other nodes

4 328 M E O'Kelly, H Kim, C Kim x 1 A 1 x 1 x 1 B A 1 2 B x 1 (a) Serial arrangement (b) Parallel arrangement E 1 ˆfx 1, x 2 g, E 1 ˆfx 1 g, E 2 ˆfx 2 g, E 3 ˆfx 1, x 2 g, R AB ˆ Pr(E 1, ˆ 0:9 0:9 ˆ 0:81. R AB ˆ Pr(E 1 Pr E 2 Pr E 1 \ E 2, ˆ 0:9 0:9 0:9 0:9 ˆ0:99. Figure 1. Computation of reliability for simple arrangements. congestion (Bell and Iida, 1997). (3) It is clear that for this particular pair A and B, 0:9 0:9 ˆ 0:81 in figure 1(a) is the joint probability of a successful linkage, thereby indicating that 81% of traffic can be carried without delay. However, the real situation is quite a bit more complicated because there are usually many paths between pairs of cities. Therefore, even if the performance capacity is set at only p ˆ 0:6 for individual linkages, the origin ^ destination-path reliability may be computed at a much higher level. The reliability is basically determined by summing up the probabilities of all disjoint events for the successful connection between origin and destination nodes: they are enumerated with the rule that they should be mutually statistically independent. The standard mathematical expression is as follows: R OD G, p ˆ Xn PrfE i g, i ˆ 1 PrfE i g ˆ Ym Prfe ij g, j ˆ 1 where R OD is the reliability for two selected nodes, origin O and destination D, G is a graph of a network with given probability p for edges, PrfE i g is the probability of the disjoint event E i, and e ij are the edges j constituting E i. In a serial arrangement [figure 1(a)] only one successful event set E 1 fx 1, x 2 g is enumerated, and the reliability is calculated by multiplying probabilities for two serial linkages. For computation in a parallel arrangement, the inclusion ^ exclusion formula is generally used to avoid double counting derived from all unions among successful statistically independent events. (4) A network in a parallel system [figure 1(b)] can survive if, and (3) The parameter p is interpreted differently by various disciplines. In systems engineering or circuit theory, p ˆ 0:9 implies the reliable state 90% of the time. Statistical mechanics stress distributions and ensemble averages which are most useful when very large datasets are considered. We adopt a definition close to the transportation-network analysis offered by Bell and Iida (1997) as discussed in the text. (4) The general expression of the inclusion ^ exclusion method is as follows (Bell and Iida, 1997, page 185): R OD G, p ˆ Xn PrfE i g Xn X PrfE i \ E t g i ˆ 1 Xn i ˆ 1 i 6ˆ t X X PrfE i \ E t \ E s g.:: 1 n 1 Prf\ i ˆ 1 n E i g. i ˆ 1 i 6ˆ t s 6ˆ i; t

5 Internet reliability with realistic peering 329 only if, at least one of the successful events, E 1 fx 1 g or E 2 fx 2 g operates, and the probability of double counting set E 1 \ E 2 ˆfx 1 x 2 g is excluded to ensure E 1 and E 2 are disjoint events with respect to each other. If a network has a mixed structure consisting of these two forms of systems, the measurement would be much more complex as attention must be paid to the overlapping events during the calculation of reliability. In this paper we utilize an algorithm that is based on a well-known Boolean algebra method and is considered to be more efficient than other exact techniques. Although the procedure is different from a general procedure of reliability computation, the Boolean algebra method generates the same result with a reduction both in the enumeration of disjoint events and in computation time (Colbourn, 1987; Dotson and Gobien, 1979; Yoo and Deo, 1988). The main idea of this algorithm is to work forward from an initial successful event to another disjoint event by finding its complement and other unused viable paths. This process continues until all disjoint events are found. For example, if there are three successful events, E 1, E 2,andE 3, then disjoint events D i by Boolean logic are generated as: D 1 ˆ E 1, D 2 ˆ E c 1 \ E 2,andD 3 ˆ E c 1 \ E c 2 \ E 3. Reliability is then computed with the summation of the probabilities of D 1, D 2,andD 3. Figure 2 illustrates how the algorithm finds the disjoint events for a parallel arrangement. The following section outlines our methodological approach to the geographical vulnerability of the Internet for US cities through reflection on the peering characteristics of Internet hubs. Example network x 1 x 1 A 1 2 B A 1 2 B x 2 E i ˆ [x 1 x 2 ] D 1 ˆ E 1 ˆ Pr[1 0] ˆ 0:9 x 1 x 1 A 1 2 B A 1 2 B x 2 x 2 R ˆ D 1 D 2 ˆ 0:99 D 2 ˆ E c 1 \ E 2 ˆ Pr[ 1 0]Pr[0 1] ˆ (1 0:9) (0:9) ˆ 0:09 Figure 2. Reliability algorithm using Boolean algebra. 3.2 Peering arrangements in a hub If two backbone providers make a peering agreement at their common node, it means that each provider or IBP allows traffic from other participants to transit its backbone, thereby increasing access across the country. These cooperative arrangements are generally established for such particular city nodes as IXs (Internet exchange points), NAPs (network access points), and MAEs (metropolitan area exchanges) by installing

6 330 M E O'Kelly, H Kim, C Kim necessary equipment to switch traffic. (5) Owing to these cooperative arrangements, each IBP can get expanded service coverage. Such system can be used as a backup to make networks more reliable via redundant paths in the case of either traffic congestion or operational problems. The function of peering arrangements in a hub is explained from a geographical perspective with reference to figure 3. Suppose that IBPs A, B, and C each take charge of only one service area, and the Chicago NAP is suggested for the sake of illustration as a single hub at which these IBPs are joined for the purpose of peering. Then, for example, the traffic originating in city W a located in MAE West can flow into city E a located in MAE East through the Chicago NAP, where the interexchange of traffic happens through mutual interconnections between participants. Two important characteristics of peering arrangements with regard to origin ^ destination network reliability are stressed in the following observations. First, the reliability of the Internet depends on the peering structure in a hub. There are two routes for successful communication between city W a and city E a as indicated in figure 3(a). If at least one of them works, the flow for two cities can be carried out. However, assuming linkages `ab' and `ac' in a given peering arrangement were simultaneously disconnected, no traffic originating in any member cities of MAE West (W a, W b,.::) could flow to any cities located in either MAE Central or MAE East owing to the combination of disconnections. Second, the network reliability can be affected by types of peering. In the case of public peering all IBPs joined in a particular hub are fully meshed with each other, thereby allowing multiple interconnections to exchange traffic for other participants. This is the model depicted in figure 3. On the other hand, private peering allows traffic to be carried through a dedicated interconnection for only two IBPs according to their bilateral contract. Therefore, any member with private peering can communicate with other members even in the case either of malfunction of public hubs or of backbone failures of nonmembers, although the member might suffer from a general degradation of network performance. (6) The Internet can retain its operational functionality by virtue of peering arrangements until the hub reaches a critical threshold. The reliability for a pair of cities could not be readily broken but, rather, the cities experience degradation of service caused by malfunctions of linkages in a hub. The idea of reliability envelope is appropriate to take into account their realities. 3.3 Reliability envelope The all-or-nothing assumption in prior research tended to lead to the analysis of survivability by showing the potential for network disruption as a consequence of complete damage at any particular node. This simplified assumption is relaxed here by taking into account all possible cases of damage generated by combinatoric rules. This research simulates the reliabilities for the set of interconnection linkages at the hub and then examines the variation of reliabilities both with the default full-connection (5) These relationships can be identified by referring to autonomous system numbers providing information on adjacencies among ISPs. These interconnections are generally agglomerated within a certain geographic area not only to improve efficiency but also to reduce cost for the establishment of interconnections. However, these very specific locations are not always known (Gorman et al, 2004; Gross, 2004) because of the proprietary nature of the data. In this study, we define this colocated area as a city node in a broad sense. (6) Currently, the role of private peering in alleviating traffic congestion is stressed. By some estimates 80% of Internet traffic was exchanged at one point. However, recent indications show that NAPs will be able to gain their former attraction again as efficient meeting points with the help of the asynchronous transfer mode and other advanced technologies (Kende, 2000).

7 Internet reliability with realistic peering 331 MAE West Geographical coverage of IBP B Chicago NAP (interconnection point) MAE East city E a city W a city W b A ab ac B C bc MAE Central city E b city E n city C a city W n city C b Geographical coverage of IBP A (a) Geographical coverage of IBP C city C n MAE West Chicago NAP (interconnection point) MAE East city E a city W a city W b ab B city E b A ac bc C MAE Central city E n city C a city W n city C b (b) city C n Figure 3. Function (a) and malfunction (b) of the peering arrangement in a hub. MAEs and NAPs constitute metropolitan area exchanges and network access points, respectively; IBP denotes Internet backbone provider. assumption and with the probabilistically controlled assumption. Particular peering arrangements in selected hubs can be modeled as a set of subnetworks, thereby influencing the whole network reliability. As the number of damaged linkages at the hub increases towards total loss, the reliability of the subnetwork is expected to fall below the tolerable level, thereby degrading the overall reliability of the network as well as reliabilities for city pairs. The reliability envelope illustrated in figure 4 (over) represents variations in reliability for a particular pair of cities on the Internet. When an attack impacts r out of n components as shown on the x-axis (n represents the number of linkages at the hubs), there is a range of consequences as shown on the y-axis. It could be that the r chosen

8 332 M E O'Kelly, H Kim, C Kim y Upper bound (maximum reliability) 1.0 Range between upper and lower bound Lower bound (minimum reliability) Reliability b The desired level of network Length of lower bound The minimum number of failures to reach total loss 0.0 nc 1 n C 2 n C 3 n C 4 n C r n C n Number of failures Figure 4. The variation of reliability on the interconnection failures in a hub. failures are highly damaging (in the worst case) or relatively unimportant (in the best case). For example, the combination of r critical linkage failures out of n in a hub causes the worst outcome at the rth stage; however, r failures consisting of noncritical interconnections could be much less damaging. As the size of r relative to n increases, the chances that r failures produce minimal damage declines, thereby giving rise to an envelope with an upper and a lower bound. The range between the best (upper) and worst (lower) bounds is shown graphically in figure 4. Some analytical points from this schematic diagram can be drawn as follows. First, the ranges between the upper and lower bounds at each of r steps show how susceptible the subnetwork is. If the network can keep a narrow range of impact even as the number of attacked components r increases, then the network retains a good resiliency to attacks. The larger the range, the more vulnerable the network. Second, the survivability of a hub network can be seen by looking at the shape of the lower bound. The length of the lower bound before reaching a complete loss of its functionality indicates how long a pair of nodes on the network or even an entire network can retain its communication ability (or survive) with an increase of attacks. The longer the lower bound extends before reaching total loss level (n failures), the more survivable the network. This point is similar to the survivability measure in previous work (Grubesic et al, 2003). Third, the desired level of network reliability can be discussed in terms of network design. What is the smallest number of interconnections required to sustain the desired reliability ( b in y-axis) against attacks as hits are taken, and which case of disconnections in a particular stage is most damaging? These research questions are examined on the basis of the reliability envelope. Two kinds of analysis are conducted and discussed in the following section. 4 Analysis In this section we introduce the analysis of empirical Internet backbone networks. The introduction is followed by three particular empirical analyses based on the networks of IBPs in the United States. 4.1 Empirical analysis The Internet network is a very complex system operated by hundreds of participants, and many peering arrangements are geographically distributed at multiple hubs.

9 Internet reliability with realistic peering 333 Because of the upheaval of IBPs in the United States during the last few years, it is difficult to obtain information on nodes and backbone linkages during an identical time span for all current IBPs. In addition, topological changes of networks, particularly peering relationships among ISP members, have changed so frequently that data choices for academic research are fairly limited. (7) Our empirical analysis is therefore conducted under several idealized assumptions; the intent is to represent the types of network scenarios that could happen, and at the same time to caution against inference beyond the scope of these hypothetical representations. (1) Selection of participants. Five major IBPs in the United States, FNS, ICG, IDT, SERVINT, and SAVVIS, are selected as of 2000 because they present comparatively clear network topologies and peering information for hubs, and they are all still in the market. The primary data was obtained from Boardwatch Magazine (2000) and websites of relevant ISPs, the IX directory provided by the TeleGeography website ( was also retrieved. As these resources were identical to those used in previous works, our analyses can be compared in the same line, even though they could show a snapshot reflecting a temporal profile of the dynamic of ISP markets. (2) Selection of hubs and peering arrangements. To ensure an accurate and more realistic situation, the public-peering structure table is constructed with the fundamental knowledge that every participant in NAPs or an MAE can exchange their traffic. A private-peering relationship is considered to be based on the known private-peering information. Three public access pointsöchicago, Dallas, and San Franciscoöare therefore selected to have a fully interconnected arrangement at each hub, and one interconnection for private peering is assumed at Dallas, which is considered to have a dedicated linkage only between ICG and SERVINT. (8) Because our analyses are focused on these idealized assumptions, all simulation results assume that other hubs and peering arrangements work without any operational problems. To examine how sensitively reliabilities vary with the probability parameter, simulations are conducted under different parameters, p ˆ 0:9 and p ˆ 0:6. On the basis of data obtained under these assumptions, there are forty-four cities in combined layers consisting of five networks. Figure 5 (over) illustrates the layer of the five select backbone networks and peering arrangements for select hubs. As a preliminary analysis we measure reliabilities for all pairs of cities under simplified assumptions (902 total cases). As it is important to know which cities show the highest reliability, the overall performance of each city is examined in terms of average reliability. Average reliability is computed by summing the reliabilities from a particular city to the other cities excluding itself. Table 2 (over) shows average reliability rankings of thirty cities using both values of p. The range of reliability response is highly sensitive to the value of the parameter. For example, the small reliability range (0.1117) between (7) For example, the peering relationship in the Chicago NAP has been changed both in terms of its participants and even in terms of number. During the last two years only twenty-one companies out of forty-six kept their memberships (SBC, 2002). MAEs have also experienced membership changes. (8) The first reason for selecting these three hubs is because they serve as common nodes among five participants, according to our available data. Another reason is to reduce the computation time, because even our simplified model requires 2 15 ˆ (n ˆ 15, the number of interconnections at the hubs) reliability computations per city pair. Third, if even our simplified model shows the network to be fairly resilient, then it may be seen as a conservative estimate of the Internet in the real world, as the current Internet has many more hubs than our model. Our results indicate the benefits of redundant multiply connected hubs. (We are grateful to a referee for this observation.)

10 334 M E O'Kelly, H Kim, C Kim SAVVIS ICG IDT FNS SERVINT Chicago NAP MAE Central San Francisco NAP IDT SERVINT ICG SAVVIS FNS Figure 5. Five Internet backbone providers in empirical analysis. highest and lowest city node is found at p ˆ 0:9, whereasatp ˆ 0:6 the gap ( ) is larger and differences among city nodes are easily identified to two decimal places. In particular, the less reliable city nodes critically degraded with the change of parameter as shown in table 2(b). Because those city nodes rely on just a few disjoint paths to connect the other city nodes, their reliabilities respond sensitively to the parameter value. In table 2(a) Chicago appears to be the city with the highest reliability and the city with the lowest reliability is revealed as Nashville, on the basis of both parameters. What is significant is that major NAPs such as Chicago, Atlanta, Dallas, Los Angeles, and New York are ranked among the ten most reliable cities. This group, plus two other NAP points, San Francisco and Philadelphia, show reliabilities above for p ˆ 0:9. This result is consistent with previous results based on survivability (O'Kelly and Grubesic, 2002; Grubesic et al, 2003), thereby implying that these hub location cities would remain well connected with the other cities or vice versa. Conversely, the current Internet depends highly on selective hubs, such that its telecommunication abilities can be critically degraded if simultaneous malfunctions happened at these important cities. It is also important to note that nonmajor hubs, such as Cleveland, Cincinnati, and Columbus, are entered among the ten most reliable cities. The tier showing lowest reliability, with R < 0:9 at p ˆ 0:9 or R < 0:6 at p ˆ 0:6, comprises only five cities; Pittsburgh, Dayton, Fort Worth, Birmingham, and Nashville. A total of eighteen pairs of cities, mainly combined among seven cities (Atlanta, Chicago, Dallas, Denver, New York, St Louis, and Washington, DC), are found as the most reliable pairs whereas the lowest pairs in terms of reliability are composed of combinations from the five cities with the lowest reliability, listed in table 2(b). (9) (9) The four lowest reliability city pairs are Birmingham ^ Nashville, Fort Worth ^ Nashville, Dayton ^ Nashville, and Pittsburgh ^ Nashville.

11 Internet reliability with realistic peering 335 Table 2. (a) Average reliability ranking for the top fifteen city nodes under the all-or-nothing assumption. (b) Average reliability ranking for the bottom fifteen city nodes under the all-or-nothing assumption. Rank City node State Average reliability p ˆ 0:9 p ˆ 0:6 (a) 1 Chicago Illinois Atlanta Georgia Cleveland Ohio Cincinnati Ohio Denver Colorado Dallas Texas Columbus Ohio Los Angeles California New York New York Houston Texas Miami Florida Richmond Virginia San Francisco California Detroit Michigan San Jose California (b) 30 San Antonio Texas Fort Wayne Indiana Norfolk Virginia Akron Ohio Portland Oregon Kansas City Missouri Hackensack New Jersey Corpus Christi Texas Louisville Kentucky Charlotte North Carolina Pittsburgh Pennsylvania Dayton Ohio Fort Worth Texas Birmingham Alabama Nashville Tennessee The reliability potential map, based on average reliability for forty-four city nodes, in figure 6 (over) indicates where either potentially vulnerable or reliable cities are located under normal conditions. A characteristic geographic distribution in the United States is highlighted, with relatively susceptible areas formed around lower ranked city nodes, and more reliable areas near major hub cities that are highly ranked [table 2(a)]. (10) 4.2 Network reliability with disconnections in three hubs From the preliminary analysis we can choose three cases in which to examine the network reliability variation under the circumstance of closing off interconnections in three selected hubs in Chicago, San Francisco, and Dallas. The first tested simulation is between the least reliable city pair, Pittsburgh ^ Nashville. The second case, one of the most interesting city pairs, is the Seattle ^ Boston pair, geographically the most distant pair out of all those possible in our model. The third set of simulations focus on the reliability envelopes for selected city pairs containing Los Angeles, which prior research (10) Most city pairs experience a reliability improvement in the combined layer as a consequence of these exchange points. In the case of Seattle ^ Boston, the reliability of in the SERVINT network is significantly improved to after combining with other networks.

12 336 M E O'Kelly, H Kim, C Kim N Reliability High : miles Low : Figure 6. Reliability potential map. has shown to be a city whose loss would prove quite lethal in term of survivability of the whole Internet (Grubesic et al, 2003). Geographic patterns of reliabilities from Los Angeles to the rest of the city nodes for selected stages is explored. The simulation results in this section are presented by applying parameter p ˆ 0: The lowest reliability city pair: Nashville öpittsburgh Nashville and Pittsburgh can be wired into the whole Internet backbone network by virtue of the networks of ICG and SAVVIS, respectively. Recalling the characteristics of accessibility measurement, they are in the least accessible group with respect to the total accessibility or connectivity because they show the smallest degree of node (the number of linkages directly connected to the node, dn ˆ 1) both as individual networks and even in a combined network. In the case of the all-or-nothing state when five networks are combined into one layer, the reliability of this city pair shows öhowever, they could become more reliable, measured at , if no attack on the hubs were supposed. This is because interconnections of subnetworks could make the whole network more resilient by increasing alternative routes in hubs. Reliability falls to in the case of closing off fifteen total interconnections at three hubsöa very small degradation ( ) from the normal condition. This indicates that the two cities could still maintain the communicative ability with a probability of even if simultaneous operational problems occurred at the Chicago, Dallas, and San Francisco hubs. At first glance this simulation shows extremely small degradation. However, this can be justified not only because there exists a considerable number of redundant paths between the two cities created by overlaying five individual networks, but also because hub cities become more resistant as a result of the complexity of subnetworks in a hub. (11) A reliability envelope with the three types of variation illustrated in figure 7(a) depicts how reliabilities would change for all possible combinations of disconnections at each stage. The narrow gap between the maximum and the average by the third stage shows a good possibility of occurrence of the maximum reliability, and the upper (11) When simulated with a different parameter ( p ˆ 0:6), the reliability of this city pair dropped considerably to , but the degradation was measured as a relatively small gap ( ), thereby proving the resiliency of the three hubs.

13 Internet reliability with realistic peering Maximum Average Minimum Reliability Reliability Reliability (a) (b) (c) The number of disconnections in the three hubs Figure 7. Reliability envelopes for selected city pairs. (a) Nashville ^ Pittsburgh, (b) Seattle ^ Boston, (c) Los Angeles ^ Nashville. bound might retain its highest reliabilities up to the eleventh stage. At the eleventh stage, however, the maximum reliability is unlikely to happen as the gap between the maximum and the average is much wider than that between the average and the lower bound. The upper bound significantly drops from the twelfth stage, which indicates that a best case could rarely occur in the subsequent stages. The lower bound, the worst

14 338 M E O'Kelly, H Kim, C Kim scenario, quickly reaches the state of the lowest reliability at just the fifth stage, after experiencing significant initial degradation. It is worthwhile to examine the combinations of disconnections causing the worst scenarios for each stage. For example, the lowest reliability at the third stage comprises three disconnections all from the peering arrangement of the Chicago hub. Likewise, Chicago is revealed as the main cause of the lowest reliabilities for all other stages, suggesting that a malfunction of the Chicago hub could lead to more severe impacts on the reliability degradation than the other two hubs. Therefore, Chicago is regarded as the most significant hub for the communication between these city nodes The most distant city pair: Seattle ^ Boston Although these two cities are geographically far apart, it is surprising that both cities are equally ranked as sixteenth in average reliability (R ˆ 0: ), and the city-pair reliability is also highly positioned as 147th of 946 total city pairs. These results can be explained by the increase of disjoint paths and the resilience of the three hubs. First, by combining five networks into one whole network with peering, all individual networks having this city pair could experience a reliability improvement. For example, the reliability of Seattle ^ Boston in the SERVINT (R ˆ 0:8874) and ICG (R ˆ 0:8894) networks are significantly improved to R ˆ 0:99999 in the combined network. This is mainly because this city pair could have a higher capability of alternative routings as a result of the increased disjoint paths owing to peering arrangements. In particular, this city pair is measured to have fifty disjoint events, which is greater than other cases such as Birmingham ^ Nashville (13), Nashville ^ Pittsburgh (43), and Los Angeles ^ Nashville (36). Second, the strong communication ability of this city pair is shown by the reliability envelope in figure 7(b). The first breaking point of the upper bound does not occur before the fourteenth stage, thus maintaining the initial upper reliability. At the lower bound no reliability changes are observed after very small degradation at the first stage. The capability, which sustains very narrow gaps between upper and lower bound until the end of all stages, highlights the strong resilience of this city pair with respect to the possible damage of the three hubs. The gradual decline of average reliability indicates that this pair would be less vulnerable and more stable than the Nashville ^ Pittsburgh pair. The reason for the abrupt decrease of reliability in the worst-case scenario (shown in the lower bound at the first stage) is from disconnections in the Chicago hub. In addition, all other worst cases at the lower bound also result from the combinations of disconnections in the peering arrangement of the Chicago hub. On the basis of these facts, the role of the Chicago hub in retaining the communication ability is important, as it was in the Nashville ^ Pittsburgh case. Accordingly, it is worth noting that the disadvantage of the geographical location of both cities should be mitigated as a result of the intensified network reliability from the resilience of peering arrangements in the three hubs Reliability from Los Angeles to the other cities The reliability ranking for Los Angeles to the other forty-three city nodes is revealed as being quite similar to the result of average reliabilities. For example, twelve city nodes in the top fifteen in terms of average reliability in table 2(a) show the highest reliability with Los Angeles (R > 0:999999, p ˆ 0:9) and Washington, DC, Austin, TX, and Boston become newly entered in the top fifteen. In addition, the five lowest reliability city nodes, Birmingham, Fort Worth, Dayton, Pittsburgh, and Nashville, make up the bottom five in communication with Los Angeles. It is interesting to examine which city nodes are expected to experience degradation in the case of disruptions to the three hubs. Table 3 shows the top ten city nodes in terms of reliability degradation after disruption to the three hubs, with p ˆ 0:9.

15 Internet reliability with realistic peering 339 Table 3. Reliability degradation from Los Angeles to select city nodes ( p ˆ 0:9). Figure in parenthesis is the ranking according to each measure. City node State Condition of the three hubs Degradation no failure three hub failures Chicago Illinois (1) (26) (3) Dallas Texas (1) (19) (4) Newark New Jersey (22) (20) (10) Akron Ohio (28) (33) (2) San Antonio Texas (28) (28) (5) San Francisco California (30) (38) (1) Hackensack New Jersey (31) (29) (9) Corpus Christi Texas (36) (35) (6) Charlotte North Carolina (37) (36) (8) Nashville Tennessee (43) (43) (7) San Francisco is revealed as experiencing the most severe telecommunication degradation with Los Angeles among forty-three city nodes, because of its geographic dependence on and closeness to Los Angeles. The degradations of Akron, San Antonio, and Corpus Christi are due to geographic closeness to their adjacent hubs, such as Chicago and Dallas. For example, most paths to reach Akron from Los Angeles should pass through the Chicago hub. According to table 3 our interest lies on two select cases which are investigated in order to examine reliability variations: (1) the lowest reliability (Los Angeles ^ Nashville; (2) the highest in normal condition, but with considerable degradation after attacks on the three hubs (Los Angeles ^ Chicago). With attention paid to the case of the lowest reliability in figure 7(c), a stepped decline of average reliability is expected with the increase of disconnections between Los Angeles and Nashville. The reasons for significant drops of the lower bound shown at the first, second, and third stages are revealed by examining the combinations of disconnections in the three hubs. The combinations causing the worst case are (Chicago, Dallas, San Francisco: 1, 0, 0) for the first stage, (1,1, 0) for the second stage, and (2,1, 0) for the third stage, respectively. The Los Angeles ^ Chicago pair would experience considerable reliability degradations according to our simulation. This is because Chicago is a hub directly affected by disconnections in itself. Likewise, the reason for degradation both at Dallas and at San Francisco can be explained in this manner. In the case of Chicago five reliability envelopes are examined because Chicago has five access points constituting the intrahub subnetwork. Figure 8 (over) illustrates how the reliability of each nodal point varies with disconnections and highlights various envelope patterns from inner node 1 to node 5. The best envelope is found in the fifth node, in which the highest telecommunication ability under any circumstances of disconnections would be guaranteed. Various types of degradation patterns, such as concave (first and second nodes) and step-like patterns (third and fourth nodes) stress that telecommunication abilities between Los Angeles and the other cities should depend on which combinations of disconnections occur as well as which inner nodes of a hub are used as bridges to connect cities. The reliability potential maps in figure 9 (over) illustrate the change of average reliabilities from Los Angeles to other city nodes for the following selected stages: no disconnection, with five disconnections, with ten disconnections, and with fifteen disconnections (malfunction of the three hubs). According to the maps the areas with the higher reliabilities are reduced in size and the size of the lower reliability areas increases as the number of disconnections increases. This fact indicates that the

16 Reliability Node The number of disconnections in the three hubs Reliability Node 4 Reliability The number of disconnections in the three hubs Figure 8. Reliability envelopes for Los Angeles ^ Chicago (five hub nodes). Node The number of disconnections in the three hubs Reliability Reliability Node 5 Maximum Average Minimum The number of disconnections in the three hubs Node The number of disconnections in the three hubs 340 M E O'Kelly, H Kim, C Kim

17 Internet reliability with realistic peering 341 Reliability High : Low : (a) miles Reliability High : Low : (b) Reliability High : Low : Reliability High : Low : (c) (d) Figure 9. Change in average reliability from Los Angeles to other city nodes with (a) no disconnection, (b) five disconnections, (c) ten disconnections, (d) fifteen disconnections. communication ability of Los Angeles to the other cities would become vulnerable only gradually and slowly. Such city nodes as Houston, Austin, Atlanta, Denver, Columbus, and St Louis, ranked in the top ten, could maintain higher reliability, even at the last stage, but the degradation of other cities influences the overall pattern of potential reliability into less reliable phases. In particular, the areas in California would be influenced by the condition of the San Francisco hub; likewise, the reliability of some city nodes around Chicago depends considerably on the functionality of the Chicago hub. East Coast areas, beginning from Boston, through New York to Miami, appear to be less vulnerable to communication failure with Los Angeles in the event of degradation of the three hubs, because this area is relatively isolated from the direct influence of the three hubs owing to its geographic location, and also because there are many redundant paths. In contrast, city nodes forming a band in the traditional American manufacturing belt (Akron, Pittsburgh, Lousiville, Nashville, and Birmingham) are less reliably connected than other cities even under normal conditions. Of course, the result of geographical variations might come out differently on the basis of the geographic coverage of networks, the number of hubs, and the locations that are reflected in the simulation model; however, our results represent a probable scenario in this simplified model of the Internet. 5 Conclusions The analysis of network reliability from a geographic perspective illustrates several important points. The key difference between this paper and previous work measuring the vulnerability of the Internet is the transition from deterministic to probabilistic approaches. One contribution of this paper is that peering arrangements at hub cities

18 342 M E O'Kelly, H Kim, C Kim (known to be critical points in the Internet) are considered in a model which measures the performance of the network. By embedding this characteristic into a reliability measure we refine previous work done under simple assumptions and given more realistic results from a geographic perspective. In order to better examine the variation of performance with increasing disconnection, the concept of a reliability envelope is devised. According to the empirical analyses, resiliency benefits not only from peering arrangements but also from spatially distributed hubs. These cooperative interrelationships mean that the Internet is more reliable in normal status than is indicated by previous theoretical studies (Gorman et al, 2004). At the same time, the critical role of major hub cities such as Chicago, Dallas, and Atlanta in sustaining communicative abilities among cities in the United States must be emphasized. Failures in these major hub cities would significantly influence adjacent areas and could cause severe degradation for the entire network with loss of resiliency below tolerable levels. An important finding is that various patterns in the reliability envelope can be influenced by factors such as network structures, hub structures, and geographical locations. Overall geographical patterns such as a reliability potential map vary as the number of disconnections or attacks increases. Because probabilistic approaches have proven more useful than deterministic approaches, further fruitful extensions should be applied in the geographical analysis of network reliability. Several potential extensions from our current analyses are raised for future work. First, more efficient algorithms to reduce computation time and iterations should be allowed for the consideration of multiple hubs. Our analysis considers only three hubs; clearly, consideration of more peering arrangements would make more realistic simulations possible. Second, because reliability measurement is a powerful tool with which to assess the performance of a network, extensions of the analysis to evaluate overall network measures such as `reachability' or `all-terminal reliability' should be introduced in order to compare the performance between IBPs. Finally, the model can be made more realistic by taking into account influential factors such as the capacity both of linkages and of peering interconnections in hubs. The telecommunication ability of the Internet is limited by capacity factors as well as by the probability of maintaining connections during link and node failures. We hope to address these issues as well as other geographical questions in future studies. Acknowledgements. An SBC Faculty Fellowship to Dr O'Kelly that supported Hyun Kim and Changjoo Kim is gratefully acknowledged. References Albert R, Baraba si A L, 2002, ``Statistical mechanics of complex networks'' Reviews of Modern Physics 74 47^97 Albert R, Jeong H, Barabäsi A L, 2000, ``Error and attack tolerance of complex networks'' Nature ^ 382 Ball M O, Golden B L, Vohra R V, 1989, ``Finding the most vital arcs in a network'' Operations Research Letter 8 73 ^ 76 Baran P, 1964, ``On distributed communication networks'' IEEE Transactions on Communications 12 1^9 Bell M G H, Iida Y, 1997 Transportation Network Analysis (John Wiley, New York) Boardwatch Magazine 2000 Boardwatch Magazine's Directory of Internet Service Providers 12th edition (Penton Media, Cleveland, OH) Broadband Week 2001, ``Tunnel vision?ötrain wreck sounds warning bell for fiber nets'', news fiber.htm Callaway D S, Newman M E J, Strogatz S H, Watts D J, 2000, ``Network robustness and fragility: percolation on random graphs'' Physical Review Letters ^ 5471 Coburn A W, Spence R J S, Pomonis A, 1994 Vulnerability and Risk Assessment Cambridge Architectural Research Limited, 25 Gwydir Street, Cambridge CB1 2LG Colbourn C J, 1987 The Combinatorics of Network Reliability (Oxford University Press, New York)

19 Internet reliability with realistic peering 343 Curley H W, Sha D Y, 1982, ``Most vital links and nodes in weighted networks'' Operations Research Letters ^ 160 Dotson W P, Gobien J O, 1979, `À new analysis technique for probabilistic graphs'' IEEE Transactions on Circuits and Systems ^865 Gorman S P, Schintler L, Kulkarni R, Stough R, 2004, ``The revenge of distance: vulnerability analysis of critical information infrastructure'' Journal of Contingencies and Crisis Management ^ 63 Gross S W, 2004, ``MCI white paper: MAE services version 1.0'', WP8632.a.MAE%20Services.pdf Grubesic T H, O'Kelly M E, Murray A T, 2003, `À geographic perspective on commercial Internet survivability'' Telematics and Informatics ^ 69 Huston G, 1999 The ISP Survival Guide (John Wiley, New York) Kende M, 2000, ``The digital handshake: connecting Internet backbones'', WP 32, The Federal Communications Commission Office of Plans and Policy, Washington, DC Malecki E J, 2002, ``The economic geography of the Internet's infrastructure'' Economic Geography ^ 424 Malecki E J, Gorman S P, 2001, ``Maybe the death of distance, but not the end of geography: the Internet as a network'', in Worlds of E-Commerce: Economic Geographical and Social Dimensions Eds T R Leinbach, S D Brunn (John Wiley, New York) pp 87 ^ 105 Moss M L, Townsend A, 2000, ``The Internet backbone and the American metropolis'' The Information Society Journal ^ 47 NYSDPS, 2002, ``Network reliability after 9/11; a staff white paper on local telephone exchange network reliability'', New York State Department of Public Service, Office of Communications, 3 Empire State Plaza, Albany, NY O'Kelly M E, Grubesic T H, 2002, ``Backbone topology, access, and the commercial Internet, 1997 ^ 2000'' Environment and Planning B: Planning and Design ^ 552 SBC, 2002, ``MLPA member list'', Shake T H, Hazzard B, Marquis D, 1999, `Àssessing network infrastructure vulnerabilities to physical layer attacks'', paper prepared for the National Information Systems Security Conference, Arlington, VA, Shier D R, 1991 Network Reliability and Algebraic Structures (Clarendon Press, Oxford) Wheeler D, O'Kelly M E, 1999, ``Network topology and city accessibility of the commercial Internet'' The Professional Geographer ^339 YooY B, Deo N, 1988,``A comparison of algorithms for terminal-pair reliability'' IEEE Transactions on Reliability ^215 Yook S H, Jeong H, Baraba si A L, 2001, ``Modeling the Internet's large-scale topology'' Proceedings of the National Academy of Sciences of the United States of America ^ 13386

20 ß 2006 a Pion publication printed in Great Britain