Understanding the Internet AS-level Structure

Transcription

1 UNIVERSITY OF CALIFORNIA Los Angeles Understanding the Internet AS-level Structure A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Computer Science by Ricardo V. Oliveira 2009

2

3 The dissertation of Ricardo V. Oliveira is approved. Yingnian Wu Beichuan Zhang Mario Gerla Songwu Lu Leonard Kleinrock Lixia Zhang, Committee Chair University of California, Los Angeles 2009 ii

4 To my Parents... for their unconditional love and affection iii

5 TABLE OF CONTENTS 1 Introduction Background Internet Routing Inter-domain Connectivity and Peering Ground Truth vs. Observed Map Topology Liveness and Completeness Problems A Solution to the Liveness Problem An Empirical Model of Observed Topology Dynamics Data Sets An Empirical Model Comparison with router configuration files from a Tier Comparison with Internet Registry Data Evaluation of Traceroute Data Applications More Accurate View of the Topology Evaluating Theoretical Models Characterizing Evolution Trends Discussion iv

6 5 Quantifying the Topology (in)completeness Data Sets Establishing the Ground Truth Case studies Tier-1 Network Tier-2 Network Abilene and Geant Content provider Simple stubs Completeness of the public view Public view vs. ground truth Network Classification Coverage of the public view Discussion Path Exploration and Internet Topology BGP Path Exploration Methodology and Data Set Data Set and Preprocessing Clustering Updates into Events Classifying Routing Events Comparing AS Paths Characterizing Events v

7 6.3.1 The Impact of Unstable Prefixes Policies, Topology and Routing Convergence MRAI Timer The Impact of Policy and Topology on Routing Convergence Origin of Fail-down Events Impact of Fail-down Convergence Prefix Hijacking and Internet Topology Prefix Hijacking Hijack Evaluation Metrics Evaluating Hijacks Simulation Setup Characterizing Topological Resilience Factors Affecting Resilience Prefix Hijack Incidents in the Internet Case I: Prefix Hijacks by AS Case II: Prefix Hijacks by AS Discussion Related Work Internet Topology Modeling Path Exploration Prefix Hijacking vi

8 9 Conclusion References vii

9 LIST OF FIGURES 2.1 Route propagation A sample IXP. ASes A through G connect to each other through a layer-2 switch in subnet / A set of interconnected ASes, each node represent an AS. (a) shows an example of hidden links, and (b) an example of invisible links Observing Topology Over Time Number of links captured by different sets of monitors Number of monitors in RouteViews and RIPE-RIS combined Number of links, Tier-1 monitor with different starting times Visible links seen by all monitors Link disappearance period Link disappearance period, by all monitors Observation period as a function of confidence level for links Node birth from RIR Link birth from IRR Link death from IRR Visible links in Skitter, λ = , b = Comparison between routers config files connectivity and BGP data (cumulative) from a Tier-1 network Comparison of appearance times between routers config files and BGP data of a Tier-1 network viii

10 4.14 Link disappearance period, by Skitter, µ = , d = Comparison of appearance timestamps between Skitter and BGP Comparison of disappearance timestamps between Skitter and BGP Number of reachable addresses in Skitter destination list Trade-off between liveness and completeness for topology snapshot Fraction of multi-homed customers Attachment probability distribution for a target node degree Model evaluation Node net growth Link net growth Net growth of node wirings Frequency of link changes Number of collected links in DIMES Diurnal pattern of new link appearances Weekly pattern of new link appearances Link growth for Abilene (AS11537) Output of show ip bgp summary command Configuring remote BGP peerings. R 0 and R 2 are physically directly connected, while R 1 and R 3 are not Connectivity of the Tier-1 network (since 2004) Connectivity of the Tier-1 network (since 2007) Capturing the connectivity of the Tier-1 network through table snapshots and updates ix

11 5.6 Tier-2 network connectivity Capturing Tier-2 network connectivity through table snapshots and updates Abilene connectivity Projection of the number of peer ASes of a representative content provider Customer-provider links can be revealed over time, but downstream peer links are invisible to upstream monitors Distribution of number of downstream customers per AS Example of a prefix hijack scenario where AS2 announces prefix p belonging to AS1. Because of the invisible peer link AS2 AS3, the number of ASes affected by the attack is underestimated Path exploration triggered by a fail-down event CCDF of inter-arrival times of BGP updates for the 8 beacon prefixes as observed from the 50 monitors Difference in number of events per [monitor,prefix] for T =2 and 8 minutes, relatively to T =4 minutes, during one month period Event taxonomy Usage time per ASPATH-Prefix for router , Jan Validation of path preference metric Comparison between C correct,c equal and C wrong of length, policy and usage time metrics for (a) T up and (b) T down events of beacon prefixes Comparison between accuracy of length, policy and usage time metrics Number of T down events per monitor x

12 6.10 Duration of events for January Duration of events for February Number of Updates per Event, January Number of Unique Paths Explored per Event, January Duration of events for unstable prefixes, January Duration of events for stable prefixes, January Determining MRAI configuration Duration of T down events as seen by monitors at different tiers Number of unique paths explored during T down as seen by monitors at different tiers Topology example Duration of T down events observed and originated in different tiers Number of paths explored during T down events observed and originated in different tiers Median of duration of T down events observed and originated in different tiers Number of T down events over time Case where T down convergence disrupts data delivery Hijack scenario Distribution of node resilience Resilience of nodes in different tiers Understanding resilience of tier-1 nodes Resilience of nodes with different number of Tier-1 providers xi

13 7.6 Case study: AS as false origin Case studies with AS-9121 as false origin xii

14 LIST OF TABLES 4.1 Model Parameters Comparison Between Stub and Transit changes IXP membership data, July Connectivity of stub networks Coverage of BGP monitors Coverage of BGP monitors for different network types Event Statistics for Jan 2006 (31 days) Event Statistics for Feb 2006 (28 days) T down Events by Origin AS xiii

15 ACKNOWLEDGMENTS First and foremost, I would like to acknowledge my dissertation advisor Dr. Lixia Zhang for her constant support and guidance through out my dissertation. I would also like to acknowledge Dr. Mohit Lad for his infinite patience, helpful discussions and relentless support. I am also grateful to Dr. Beichuan Zhang for contributing to the original idea of modeling topology evolution by a birth/death model, Dr. Walter Willinger and Dr. Dan Pei for their guidance during the AT&T internship, Dr. Christophe Diot for his guidance during Thomson internship and Dr. Qingming Ma for his supervision while at Juniper Networks. I would like to extend a special note of thanks to Verra Morgan for her time and support during my Ph.D. Finally, various friends and colleagues have played an important role during my Ph.D., notable among them are Dr. Vasilis Pappas, Dr. Dan Massey, Rafit Izhak-Ratzin, Cesar Marcondes, Cassio Lopes, Niko Palaskas, Bruno Miranda, Leonardo Alves and my sister Raquel Oliveira. Finally, I would like to acknowledge the portuguese Fundacao para a Ciencia e Tecnologia (FCT) for their scholarships under which my Phd work was supported. xiv

16 VITA 1978 Born, Povoa de Varzim, Portugal 2001 B.E. Electrical Engineering, Faculty of Engineering of Porto University, Portugal Software developer, Oberonsis, Portugal Telecommunications Engineer, TMN, Portugal M.Sc. Computer Science, University of California, Los Angeles Intern at AT&T Labs Research Intern at Thomson, Paris Intern at Juniper Networks. PUBLICATIONS 1. Ricardo Oliveira, Dan Pei, Walter Willinger, Beichuan Zhang, Lixia Zhang, The (in)completeness of the Observed Internet AS-level Structure, to appear in IEEE/ACM Transactions on Networking 2. Ricardo Oliveira, Beichuan Zhang, Dan Pei, Lixia Zhang, Quantifying Path Exploration in the Internet, to appear in IEEE/ACM Transactions on Networking, June xv

17 Italo Cunha, Fernando Silveira, Ricardo Oliveira, Renata Teixeira, Christophe Diot, Uncovering Artifacts of Flow Measurement Tools,Passive and Active Measurement Conference, April He Yan, Ricardo Oliveira, Kevin Burnett, Dave Matthews, Lixia Zhang, Dan Massey, BGPmon: A real-time, scalable, extensible monitoring system, Cybersecurity Applications and Technologies Conference for Homeland Security (CATCH), March Ricardo Oliveira, Fernando Silveira, Renata Teixeira, Christophe Diot, The elusive Effect of Routing Dynamics on Traffic Anomalies, Technical Report, Thomson, CR- PRL Ricardo Oliveira, Dan Pei, Walter Willinger, Beichuan Zhang, Lixia Zhang, Quantifying the Completeness of the Observed Internet AS-level Structure, Technical Report, UCLA CS Department, TR , September Ying-Ju Chi, Ricardo Oliveira, Lixia Zhang, Cyclops: The Internet AS-level Observatory, ACM SIGCOMM Computer Communication Review (CCR), October Ricardo Oliveira, Ying-Ju Chi, Mohit Lad, Lixia Zhang, Cyclops: The Internet AS-level Observatory, NANOG 43, Brooklyn, New York, June 2008 xvi

18 9. Ricardo Oliveira, Dan Pei, Walter Willinger, Beichuan Zhang, Lixia Zhang, In Search of the elusive Ground Truth: The Internet s AS-level Connectivity Structure, ACM SIGMETRICS, Annapolis, USA, June Ricardo Oliveira, Mohit Lad, Beichuan Zhang, Lixia Zhang, Geographically Informed Inter-Domain Routing, in IEEE ICNP, Beijing, China, October Mohit Lad, Ricardo Oliveira, Dan Massey, Lixia Zhang, Inferring the Origin of Routing Changes using Link Weights, in IEEE ICNP, Beijing, China, October Ricardo Oliveira, Beichuan Zhang, Lixia Zhang, Observing the Evolution of Internet AS Topology, in ACM SIGCOMM, Kyoto, Japan, August Ricardo Oliveira, Ying-Ju Chi, Ioannis Pefkianakis, Mohit Lad, Lixia Zhang, Visualizing Internet Topology Dynamics with Cyclops, in ACM SIGCOMM (poster session), Kyoto, Japan, August Mohit Lad, Ricardo Oliveira, Beichuan Zhang, Lixia Zhang, Understanding the Resiliency of Internet Topology Against False Origin Attacks, in IEEE/IFIP DSN,Edinburgh, UK, June Ricardo Oliveira, Beichuan Zhang, Dan Pei, Rafit Izhak-Ratzin, Lixia Zhang, Quantifying Path Exploration in the Internet, ACM SIGCOMM/USENIX Internet Measurement Conference(IMC), Rio de Janeiro, Brazil, October Mohit Lad, Ricardo Oliveira, Beichuan Zhang, Lixia Zhang, Understanding the xvii

19 Impact of Prefix Hijacks in Internet Routing, ACM SIGCOMM (poster session), Pisa, Italy, September Beichuan Zhang, Vamsi Kambhampati, Daniel Massey, Ricardo Oliveira, Dan Pei, Lan Wang, Lixia Zhang A Secure and Scalable Internet Routing Architecture (SIRA), ACM SIGCOMM (poster session), Pisa, Italy, September Ricardo Oliveira, Mohit Lad, Beichuan Zhang, Dan Pei, Daniel Massey, Lixia Zhang, Placing BGP Monitors in the Internet, Technical Report, UCLA CS Department, TR , May Vidyut Samanta, Ricardo Oliveira, Advait Dixit, Parixit Aghera, Petros Zerfos, Songwu Lu, Impact of Video Encoding Parameters on Dynamic Video Transcoding, in IEEE COMSWARE, Delhi, India, January Ricardo Oliveira, Rafit Izhak-Ratzin, Beichuan Zhang, Lixia Zhang, Measurement of Highly Active Prefixes in BGP, in IEEE GLOBECOM, St. Louis, USA, November xviii

20 ABSTRACT OF THE DISSERTATION Understanding the Internet AS-level Structure by Ricardo V. Oliveira Doctor of Philosophy in Computer Science University of California, Los Angeles, 2009 Professor Lixia Zhang, Chair The Internet is a vast distributed system consisting of a myriad of independent networks interconnected to each other by business relationships. The border gateway protocol is the glue that keeps this structure connected. Characterizing and modeling the Internet topology is important to our understanding of Internet routing and its interplay with technical, economic and social forces. In this thesis we address several challenges that emerge when studying the Internet connectivity. First, not all the observed changes in connectivity correspond to actual changes in the topology. There are changes that may be caused by transient routing dynamics while others are real topology changes. The problem of distinguishing between these two types of changes is non-trivial, and we call it the liveness problem. We propose a solution to this problem based on a birth/death model of observed links. This solution allows to accurately detect the permanent changes in the Internet topology graph and measure topology dynamics in an accurate way. The second problem in obtaining accurate topology models is the completeness problem, which consists in establishing how much of the real topology is missing from the observed data. We address the completeness problem by defining some bounds on how (in)complete is the graph provided by the current observation. The results using ground truth information obtained from a Tier-1 ISP inxix

21 dicate that the observed Internet graph contains most of the customer-provider links, but may be missing the vast majority of the peer-peer links. Finally, we study how protocol properties such as routing convergence and resilience to prefix hijack attacks depend on the connectivity and relationship between networks. We find that networks at the border of the Internet undergo more severe path exploration because of the higher number of paths available to reach other destinations. On the other hand, we show that Tier-1 networks have the fastest convergence time because of the limited number of alternative routes. In terms of prefix hijack attacks, we surprisingly find that Tier-1 networks at the core of the Internet are vulnerable to hijack attacks from customers because of the business nature of BGP route selection. Based on our observations, we formulate a connectivity recommendation for ISPs to increase their resiliency to these type of attacks. xx

22 CHAPTER 1 Introduction The Internet has been evolving rapidly over recent years, much like a living organism, and its topology has become more complex. Characterizing the structure and evolution trends of the Internet topology is an important research topic for several reasons. It provides an essential input to the understanding of limitations of existing routing protocols, the evaluations of new designs, as well as the projection of future needs; and it will help advance our understanding of the interplay between networking technology, the resulting topology, and the economic forces behind them. Many research projects have used a graphic representation of the Internet AS-level topology, where nodes represent entire autonomous systems (ASes) and two nodes are connected if and only if the two ASes are engaged in a business relationship to exchange data traffic. Due to the Internet s decentralized architecture, however, this AS-level construct is not readily available and obtaining accurate AS maps has remained an active area of research. A common feature of all the AS maps that have been used by the research community is that they have been inferred from either BGPbased or traceroute-based data. Unfortunately, both types of measurements are more a reflection of what we can measure than what we really would like to measure, resulting in fundamental limitations as far as their ability to reveal the Internet s true AS-level connectivity structure is concerned. While these limitations inherent in the available data have long been recognized, there has been little effort in assessing the degree of completeness, accuracy, and am- 1

23 biguity of the resulting AS maps. Although it is relatively easy to collect a more or less complete set of ASes, it has proven difficult, if not impossible, to collect the complete set of inter-as links. The sheer scale of the AS-level Internet makes it infeasible to install monitors everywhere or crawl the topology exhaustively. At the same time, big stakeholders of the AS-level Internet, such as Internet service providers and large content providers, tend to view their AS connectivity as proprietary information and are in general unwilling to disclose it. As a result, the quality of the currently used AS maps has remained by and large unknown. Yet numerous projects [37, 55, 59, 20, 31, 29, 91] have been conducted using these maps of unknown quality, causing serious scientific and practical concerns in terms of the validity of the claims made and accuracy of the results reported. Obtaining accurate and complete Internet topology data is a challenging task. First, the observed AS topology snapshots only capture a subset of the real Internet topology [27, 99, 60, 77, 96, 70]. This is referred as the completeness problem. The incompleteness of the observed AS topology stems from the fact that our main source of connectivity data are BGP routing tables (Border Gateway Protocol), and BGP was designed to propagate routing information, not AS adjacencies. In BGP, only the best path is propagated to neighbors, and not all neighbors receive all routes, therefore it s only natural that there is missing connectivity information when using a limited number of vantage points. By using ground truth information of a Tier-1 ISP, we quantify to some extent the degree of incompleteness of the observed topology. We find that current set of vantage points are able to capture the totality of customer-provider links, but as much as 90% of the peer links are still escaping from observation. The invisible peer links exist mainly between nodes at the border of the network. Second, a new problem arises when we try to measure topology changes over time: the changes in the observed topology do not necessarily reflect the changes in the 2

24 real topology and vice versa. Because the observed topology is normally inferred from routing or data paths, its changes can be due to either real topology changes or transient routing dynamics (e.g., caused by link failures or router crashes). Therefore the challenge is, given all the changes in the observed topology over time, how to differentiate those caused by real topology changes from those caused by transient routing dynamics, which we call the liveness problem. Only after solving the liveness problem can we provide empirical topology evolution data such as when and where an AS or an inter-as link is added or removed from the Internet. In this thesis we develop a solution to the liveness problem based on the analysis of available data. Our analysis shows that the effect of transient routing dynamics on the observed topology decreases exponentially over time, and the real topology changes can be modeled as the combination of a constant-rate birth process and a constant-rate death process. There are several properties of BGP that depend on the structure of the Internet topology. In this thesis we study two of these properties: path exploration and resiliency to prefix hijacks. Before declaring a destination unreachable, BGP explores all backup paths until it finds a valid one. We call this process path exploration. In order to reduce delays and data loss during routing convergence, path exploration should happen as fast as possible. We show that path exploration depends on the number of alternative paths between the source and the destination, and that nodes at the border of the network with more alternative paths will experience more severe convergence delays than nodes at the core of the network. Other protocol property(or deficiency) that depends heavily on the topology is the resiliency of a node to prefix hijacks. A prefix hijack attack happens when a network X starts announcing address space that belongs to a network Y. The end result is that a fraction of the traffic will be deviated to the false origin. In some cases the false origin can even intercept the traffic and send it back to the true origin. After conducting a set of Internet scale simulations we find that networks connected with multiple Tier-1s are the most resilient to this type of 3

25 attacks. Furthermore, we also surprisingly find that Tier-1s at the core of the network are more vulnerable to prefix hijacks launched by its customers because of the policy factor in BGP route selection. The main contributions of this dissertation can be summarized as follows. First, we formulate the topology liveness problem and propose a solution for it, this is described in Chapter 4. Second, we investigate the completeness of the observed AS topology by quantifying and explaining the reasons why AS adjacencies are missing from commonly used data sources, which is described in Chapter 5. Third, in Chapter 6 we establish the dependency between the convergence of BGP routes and the topological location of both the monitor and the origin of the routes. Lastly, in Chapter 7 we show how the resiliency of networks to prefix hijack depends on how close to the Tier-1 core each network is connected. 4

26 CHAPTER 2 Background In this section we present the relevant background on Internet routing and relationships between different networks. 2.1 Internet Routing The Internet consists of more than thirty thousand networks called Autonomous Systems (AS). Each AS is represented by a unique numeric ID known as its AS number, and may advertise one or more IP address prefixes. For example, the prefix /16 represents a range of 2 16 IP addresses belonging to AS-52 (UCLA). Internet Registries such as ARIN and RIPE assign prefixes to organizations, who then become the owner of the prefixes. Automonous Systems run the Border Gateway Protocol (BGP) [78] to propagate prefix reachability information among themselves. In the rest of the thesis, we abstract an autonomous system into a single entity called AS node or node, and the BGP connection between two autonomous systems as AS link or simply link. BGP uses routing update messages to propagate routing changes. As a path-vector routing protocol, BGP lists the entire AS path to reach a destination prefix in its routing updates. Route selection and announcement in BGP are determined by networks routing policies, in which the business relationship between two connected ASes plays a major role. AS relationships can be generally classified as customer-provider or peer- 5

27 peer 1. In a customer-provider relationship, the customer AS pays the provider AS for access service to the rest of the Internet. The peer-peer relationship does not usually involve monetary flow; The two peer ASes exchange traffic between their respective customers only. Usually a customer AS does not forward traffic between its providers, nor does a peer AS forward traffic between two other peers. For example in Figure 2.1, AS-1 is a customer of AS-2 and AS-3, and hence would not want to be a transit between AS-2 and AS-3, since it would be pay both AS-2 and AS-3 for traffic exchange between themselves. This results in the so-called valley-free BGP paths [39] generally observed in the Internet. When ASes choose their best path, they usually follow the order of customer routes, peer routes, and provider routes. This policy of no valley prefer customer is generally followed by most networks in the Internet. As we will see later, the no valley prefer customer policy plays an important role in determining the impact of prefix hijacks and hence we present a simple example to illustrate how this policy works. Figure 2.1 provides a simple example illustrating route selection and propagation. AS-1 announces a prefix (e.g /16) to its upstream service providers AS-2 and AS-3. The AS announcing a prefix to the rest of the Internet is called the origin AS of that prefix. Each of these providers then prepends its own AS number to the path and propagates the path to their neighbors. Note that AS-3 receives paths from its customer, AS-1, as well as its peer, AS-2, and it selects the customer path over the peer path thus advertising the path {3 1} to its neighbors AS-4 and AS-5. AS-5 receives routes from AS-2 and AS-3 and we assume AS-5 selects the route announced by AS- 3 and announces the path {5 3 1} to its customer AS-6. In general, an AS chooses which routes to import from its neighbors and which routes to export to its neighbors based on import and export routing policies. An AS receiving multiple routes picks 1 Sometimes the relationship between two AS nodes can be siblings, usually because they belong to the same organization. 6

28 Provider Customer Peer Peer Tier Figure 2.1: Route propagation. the best route based on policy preference. Metrics such as path length and other BGP parameters are used in route selection if the policy is the same for different routes. The BGP decision process also contains many more parameters that can be configured to mark the preference of routes. A good explanation of these parameters can be found in [41]. 2.2 Inter-domain Connectivity and Peering As a path-vector protocol, BGP includes in its routing updates the entire AS-level path to each prefix, which can be used to infer the AS-level connectivity. Projects such as RouteViews [15] and RIPE-RIS [14] host multiple data collectors that establish BGP sessions with operational routers, which we term monitors, in hundreds of ASes to obtain their BGP forwarding tables and routing updates over time. Among all the ASes, less than 10% are transit networks, and the rest are stub networks. A transit network is an Internet Service Provider (ISP) whose business is to provide packet forwarding service between other networks. Stub networks, on the 7

29 A B G C F D Layer-2 cloud IXP E Figure 2.2: A sample IXP. ASes A through G connect to each other through a layer-2 switch in subnet /24. other hand, do not forward packets for other networks. In the global routing hierarchy, stub networks are at the bottom or at the edge, and need transit networks as their providers to reach the rest of the Internet. Transit networks may have their own providers and peers, and are usually described as different tiers, e.g., regional ISPs, national ISPs, and global ISPs. At the top of this hierarchy are a dozen or so tier-1 ISPs, which connect to each other in a fully mesh to form the core of the global routing infrastructure. The majority of stub networks today multi-home with more than one provider, and some stub networks also peer with each other. In particular, content networks, e.g., networks supporting search engines, e-commerce, and social network sites, tend to peer with a large number of other networks. Peering is a delicate but also important issue in inter-domain connectivity. A network has incentives to peer with other networks to reduce the traffic sent to its providers, hence saving operational costs. But peering also comes with its own issues. For ISPs, besides additional equipment and management cost, they also do not want to establish peer-peer relationships with potential customers. Therefore ISPs in gen- 8

30 eral are very selective in choosing their peers. Common criteria include number of co-locations, ratio of inbound and outbound traffic, and certain requirements on prefix announcements [2, 1]. In recent years, with the fast growth of available content in the Internet, content networks have been keen on peering with other networks to bypass their providers. Because they have no concern regarding transit traffic or potential customers, content networks generally have an open peering policy and peer with a large number of other networks. AS peering can be realized through either private peering or public peering. A private peering is a dedicated connection between two networks. It provides dedicated bandwidth, makes troubleshooting easier, but has a higher cost. Public peering usually happens at the Internet Exchange Points (IXPs), which are third-party maintained physical infrastructures that enable physical connectivity between their member networks 2. Currently most IXPs connect their members through a shared layer-2 switching fabric (or layer-2 cloud). Figure 2.2 shows an IXP that interconnects ASes A through G using a subnet /24. Though an IXP provides physical connectivity among all participants, it is up to individual networks to decide with whom to establish BGP sessions. It is often the case that one network only peers with some of the other participants in the same IXP. Public peering has a lower cost but its available bandwidth capacity between any two parties can be limited. However, with the recent increase in bandwidth capacity, we have seen a trend to migrate private peerings to public peerings. 2.3 Ground Truth vs. Observed Map To study AS-level connectivity, we need a clear definition on what constitutes an inter- AS link. A link between two ASes exists if the two ASes have a contractual agreement 2 Note that private and public peering can happen in the same physical facility. 9

31 5 Provider Peer 4 Costumer Peer Best path p p 1 p 2 (a) (b) Figure 2.3: A set of interconnected ASes, each node represent an AS. (a) shows an example of hidden links, and (b) an example of invisible links. to exchange traffic over one or multiple BGP sessions. The ground truth of the Internet AS-level connectivity is the complete set of AS links. As the Internet evolves, its AS-level connectivity also changes over time. We use G real (t) to denote the ground truth of the entire Internet AS-level connectivity at time t. Ideally if each ISP maintains an up-to-date list of its AS links and makes the list accessible, obtaining the ground truth would be trivial. However, such a list is proprietary and rarely available, especially for large ISPs with a large and changing set of links. In this thesis, we derive the ground truth of several individual networks whose data is made available to us, including their router configurations, syslogs, BGP command outputs, as well as personal communications with the operators. From router configurations, syslogs and BGP command outputs, we can infer whether there is a working BGP session, i.e., a BGP session that is in the established state as specified in RFC 4271 [78]. We assume there is a link between two ASes if there is at least one working BGP session between them. However if all the BGP sessions between two ASes are down at the moment of data collection, the link may not 10

32 appear in the ground truth on that particular day, even though the two ASes have a valid agreement to exchange traffic. Fortunately we have continuous daily data going back for years, thus the problem of missing links due to transient failures should be negligible. When inferring connectivity from router configurations, extra care is needed to remove stale BGP sessions, i.e., sessions that appear to be correctly configured in router configurations, but are actually no longer active. We use syslog data in this case to remove the stale entries (as described in detail in the next section). We believe that this careful filtering makes our inferred connectivity a very good approximation of the real ground-truth. We denote an observed global AS topology at time t by G obsv (t), which typically provides only a partial view of the ground truth. There are two types of missing links when we compare G obsv and G real : hidden links and invisible links. Given a set of monitors, a hidden link is one that has not yet been observed but could possibly be revealed at a later time. An invisible link is one that is impossible to be observed by the given set of monitors. For example, in Figure 2.3(a), assuming that AS5 hosts a monitor (either a BGP monitoring router or a traceroute probing host) which sends to the collector all the AS paths used by AS5. Between the two customer paths to reach prefix p 0, AS5 picks the best one, [5-2-1], so we are able to observe the existence of AS links 2-1 and 5-2. The three other links, 5-4, 4-3, and 3-1, are hidden at the time, but will be revealed when AS5 switches to path [ ] if a failure along the primary path [5-2-1] occurs. In Figure 2.3(b), the monitor AS10 uses paths [10-8-6] and [10-9-7] to reach prefixes p 1 and p 2, respectively. In this case, link 8-9 is invisible to the monitor in AS10, because it is a peer link that will not be announced to AS10 under any circumstances due to the no-valley policy. Hidden links are typically revealed if we build AS maps using routing data (e.g., BGP updates) collected over an extended period. However, a new problem arises from 11

33 this approach: the introduction of potentially stale links; that is, links that existed some time ago but are no longer present. A empirical solution for removing possible stale links has been developed in [72]. To discover all invisible links, we would need additional monitors at most, if not all, edge ASes where routing updates can contain the peering links as permitted by routing policy. The issues of hidden and invisible links are shared by both BGP logs and traceroute measurements. 12

34 CHAPTER 3 Topology Liveness and Completeness Problems Because individual ASes apply private routing policies to BGP updates, generally speaking one cannot observe the complete AS topology. We denote the complete real Internet AS topology graph by G real, and the topology graph that one infers from measurement data by G obsv. The observed portion of the AS topology is a subset of the real topology, i.e., G obsv G real. Knowing how much these two topologies differ is what we term the Completeness Problem. G obsv can be constructed in multiple ways. One way is to have data collectors establish BGP sessions with a set of operational routers, which we call monitors, to obtain their BGP routing tables and updates. Another way is to have a set of vantage points send traceroute probes and then to convert the obtained router paths to AS paths 1. For example, in Fig. 3.1, at time t 0, we measure the topology from monitor A by either examining A s routing table or probing other two nodes B and C. The resulting G obsv misses one link, B-C, from G real. To study graph properties of the AS topology it is important to minimize the number of missing links. Existing efforts in this area include deploying additional monitors and incorporating data from other sources (e.g., routing registry [96]). For example, if B is also a monitor, then one can observe the existence of link B-C. 1 Different from BGP monitors, traceroute vantage points are usually end hosts. However in this thesis we term both as monitors. 13

35 Figure 3.1: Observing Topology Over Time As a direct consequence of our inability to observe the complete topology, another problem, which we call the Liveness Problem, arises when we study topology evolution over time. That is, an observed change in G obsv does not necessarily reflect a change in G real. For example, in Fig. 3.1, at time t 1, link A-C goes down due to a physical failure, but this failure does not change the contractual relationship between A and C, i.e., link A-C still exists in G real. However, the routing protocol will adapt to the failure and link A-C disappears from the observation. As a result, comparing G obsv (t 0 ) with G obsv (t 1 ), we will see one link removal (A-C) and one link addition (B-C). In another example, consider the changes from time t 2 to time t 3 in Fig D changes its service provider by switching from C to B. This is a real topology change and results in one link removal (D-C) and one link addition (D-B) in both G real and G obsv. In both cases, what we observe are changes in G obsv, and the question is how to tell which ones are real topology changes happened in G real. We use appearance and disappearance to name the addition and removal of elements (i.e., links and nodes) in G obsv respectively, and birth and death to name the addition and removal of elements in G real respectively. The liveness problem concerns 14

36 how to infer the real births and deaths from observed appearances and disappearances. More specifically, when a link or node disappears from G obsv, is it still alive in G real? When a link or node appears for the very first time, has it been alive in G real before? Answering these questions is critical to studying topology evolution, as we need to know when and where births and deaths occur in G real. The liveness problem and completeness problem are related in that solving one will help solve the other. If the liveness of links and nodes is known, we can combine observations made at different times to form a more complete topology estimate. For example, in Fig. 3.1, combining G obsv (t 0 ) and G obsv (t 1 ) will give a more complete topology at time t 1, provided that we know link A-C is still alive at time t 1. Similarly, if the complete topology is known, we will be able to differentiate real topology changes from transient routing changes. For example, if we know the complete topology in Fig. 3.1, we will not take the appearance of link B-C at time t 1 as a birth. However the liveness problem and completeness problem are also fundamentally different. On the one hand, even if we know the liveness of all the observed links and nodes over time and are able to combine observations made through a long time period, we still do not know whether the combined topology is complete, or how incomplete it may be. For example, in Fig. 3.1, from time t 2 to time t 3, knowing the liveness of links and nodes does not help tell whether link B-C exists. On the other hand, even if monitors are placed at every node to capture all the links (except those having failures at the moment), when link A-C disappears from the observation at time t 1, we still cannot tell instantly whether it is due to an operational failure or the termination of the inter-as contract, although observations over time can provide a good estimate as described later in this thesis. Both the liveness problem and completeness problem are important to a full understanding of the Internet topology and its evolution. An ideal solution would be having 15

37 all the ISPs register their inter-as connectivity at a central registry and keep their entries up-to-date, which, unfortunately, does not seem feasible in the current Internet. A near ideal solution would be placing a monitor in each AS, which is also infeasible in reality. A number of research efforts have been devoted to making G obsv more complete, without knowing exactly how close the obtained G obsv is to G real. However, to our knowledge, no one has addressed the liveness problem, which has been a major hurdle to empirical studies of topology evolution. In this thesis, we focus on the liveness problem and propose a solution based on the analysis of available topology data. Intuitively, real topology changes generally occur over relatively long time intervals (e.g., months or even years), while transient routing changes happen within much shorter periods (e.g., minutes or hours). Thus if we keep observing the topology over time, we should be able to differentiate topology changes from transient routing changes. For example, if a link disappears and re-appears after a short period of time, it is most likely that the disappearance is not a death. If a link disappears and never re-appears again over a long time period, it is most likely that the link no longer exists. The research question is how long one should wait before declaring a birth or death with a given level of confidence. We develop an empirical model that captures the effects of long-term topology changes and short-term routing changes on observed topologies. Internet topology can be abstracted at different granularity, e.g., router-level topology, AS-level topology, and ISP-level topology (a number of ISPs have multiple ASes). Although this thesis focuses on the AS-level topology, the liveness problem is a general problem that exists independently from whether the nodes in Fig. 3.1 are routers, ASes, or ISPs. Thus we believe that solving the problem at the AS-level could lead a way to liveness solutions at other granularity. For example, if we can identify real 16

38 topology changes for each AS, by combining the behavior of ASes that belong to the same ISP, we will get the topology changes for ISP-level topology. One of our future work is to apply the methodology developed in this thesis to other types of topologies. 17

39 CHAPTER 4 A Solution to the Liveness Problem In this chapter we develop a solution to the topology liveness problem based on empirical data and provide some example applications of the model. 4.1 An Empirical Model of Observed Topology Dynamics We develop the model using BGP log data, verify its consistency with information extracted from Internet registries, and evaluate the suitability of router configuration files and traceroute data sets in solving the liveness problem Data Sets We use data from four different types of sources: BGP, router configurations, traceroute, and Internet registries. The BGP data consists of both routing tables and updates collected by RouteViews [15] and RIPE-RIS [14] from a few hundreds of monitors between January 1, 2004 and December 1, 2006, a period of almost three years 1. From BGP routing tables and updates, we extract topology information (i.e., AS nodes and links) and record the timestamps of appearances and disappearances of links and nodes. There are totally 27,972 nodes and 123,182 links in the entire data set. To evaluate the effects of different monitors, we group BGP data into three sets. 1 The main reason for starting from 2004 instead of earlier is to have an adequate number of monitors for the entire measurement period. 18

40 700 Cumulative number of links observed Tier-1 Set-54 All Number of days since January 1 st 2004 Number of peers Number of covered ASes Figure 4.1: Number of links captured by different sets of monitors Figure 4.2: Number of monitors in Route- Views and RIPE-RIS combined Cumulative number of links Cumulative number of links observed Data Linear component Fit Number of days since January 1 st Number of days since Jan 1 st 2004 Figure 4.3: Number of links, Tier-1 monitor with different starting times Figure 4.4: Visible links seen by all monitors 19

41 Tier-1: data from a single monitor residing in a Tier-1 network. Set-54: data from a set of 54 monitors residing in 35 ASes; these monitors are present throughout the entire measurement period. ALL: data from all monitors. The traceroute data is collected and kindly provided to us by three research projects: Skitter [17], DIMES [82], and iplane [57]. They all have monitors around the globe to periodically traceroute thousands of destination IP addresses, and convert router paths to AS paths. They differ in the number of monitors, locations of monitors, probing frequency, and the list of destinations to probe. Both Skitter and DIMES have data from January 1, 2004 to December 1, 2006, but iplane s data collection only started from late June, Each data set comes with an AS adjacency list describing the AS topology it observes. We also extract AS number allocation data from Regional Internet Registries (RIR) [12], and AS connectivity information from Internet Routing Registries (IRR) [7]. In addition to the above publicly available data sources, we also made use of router configuration data of all the routers of a Tier-1 backbone network, which includes historical configuration files of more than one thousand routers filtered as described in [70]. Moreover, we have access to ibgp feeds of several routers in this network. Finally, in Section 4.3 we also use ibgp data provided by Abilene, the US research and educational network An Empirical Model We first use BGP data to develop an empirical model for observed topology changes. Before starting the model development, we would like to note an important difference between links and nodes in terms of their observability. Due to the relatively small 20

42 Cumulative number of links Tier-1 Set-54 All Disappearance period of links Cumulative number of links Data Linear component Fit Disappearance period Figure 4.5: Link disappearance period Figure 4.6: Link disappearance period, by all monitors number of existing monitors and the rich connectivity among ASes, many links are not seen on the first day of observation; some of them get revealed through routing dynamics over time. However, because most ASes (over 99%) originate one or more prefixes, they appear in the global routing table on the first day of observation; the small number of remaining transit ASes behave in the same way as links in terms of their observability. As a result, the same model applies to both links and nodes. We will focus on developing the model for links, and only show the results of applying the model to nodes The Appearance of Links and Nodes Observations: Fig. 4.1 shows the cumulative number of unique links captured by different monitor sets over time. Taking the Tier-1 curve for instance: on the first day, the observed links are those in the monitor s routing table on January 1, 2004; a point (200, 40000) on the curve means that during the first 200 days, this monitor has seen unique links in total from its BGP routing tables and updates. 21