Efficient Methodical Internet Topology Discovery

Transcription

1 Efficient Methodical Internet Topology Discovery Alistair King Supervisor: Dr Matthew Luckie This report is submitted in partial fulfilment of the requirements for the degree of Bachelor of Computing and Mathematical Sciences with Honours at the University of Waikato. March 12, 2013

2 c 2013 Alistair King All Rights Reserved i

3 Abstract There are a wide-ranging and diverse set of tools and techniques which are used by researchers to gain an understanding of the topological layout of the Internet. These current macroscopic Internet topology mapping techniques however, are either inefficient, do not discover the complete topology accurately or simply have not been implemented in a useful manner. In order to carry out large-scale Internet mapping projects, a system which can map Internet topologies in an accurate and methodical manner without taxing the infrastructure is required. This report outlines the implementation and testing of an existing efficient algorithm in a useful context and also the research, design, implementation and testing of a novel algorithm, doubletree load balancer, which allows complete topologies to be discovered in a way that is far more efficient than the current state of the art. Results from testing show that doubletree load balancer has over 96% link coverage when compared to load balancer traceroute while using 30% fewer probes. ii

4 Acknowledgements The author would like to thank the following people for their contributions to this project: Dr Matthew Luckie - for supervising and providing invaluable contributions and oversight to the project. WAND Group - for providing all sorts of miscellaneous help such as feedback and advice on presentations and support with L A TEX. CAIDA - for providing helpful advice and a location to work and conduct testing from. Thanks also to friends and family who have provided support in a multitude of ways, without which, this project would not have been possible. iii

5 Contents 1 Introduction Motivation Internet Topology Basics Internet Topology Structure Traceroute Load Balancing Load Balancer Traceroute The Problem Contributions Background BGP Existing Topology Discovery Systems skitter Ark Mercator Rocketfuel Summary Improving Traceroute Efficiency Doubletree Improving Traceroute Accuracy Alias Resolution iv

6 2.4.2 Paris Traceroute Improving Traceroute Completeness Load Balancer Traceroute Windowed Doubletree Summary Doubletree Load Balancer Traceroute Overview Challenges Implementation Background Doubletree Implementation Doubletree Load Balancer Implementation Doubletree Load Balancer Modes Distributed Measurement Control Server IPID Based Alias Resolution Results and Analysis Methodology Method Comparisons Doubletree Results Doubletree Load Balancer Results Sequential List Results DTLB+ Results Conclusion Contributions Future Work BGP Based Path Length Inference Handling RFC1918 Addresses in Doubletree Probing Past Per-Packet Load Balancers v

7 6.2.4 Probing Past Unresponsive Hops Bibliography 57 vi

8 List of Acronyms CAIDA Cooperative Association for Internet Data Analysis BGP DNS Border Gateway Protocol Domain Name Service ICMP Internet Control Measurement Protocol ISP IP IPID Internet Service Provider Internet Protocol Internet Protocol Identifier IPv4 Internet Protocol version 4 SDSC San Diego Supercomputer Center SSH TOS TTL UDP Secure Shell Type Of Service Time To Live User Datagram Protocol WAND Waikato Applied Network Dynamics vii

9 Chapter 1 Introduction There is fairly extensive research into the area of Internet topology discovery from which a number of tools and techniques have been developed which aim to improve on various aspects of the field. Many of the tools developed to date provide only a best-effort approach to topology discovery; that is, they make little attempt to ensure all of the topology has been discovered. This report has two main focuses. The first is the implementation of an existing traceroute improvement within a system which is currently in use in the longest running Internet topology mapping project. The second focus is the proposal, development and implementation of a new algorithm for efficiently gaining a complete and accurate map of Internet topology. 1.1 Motivation It is useful for Internet researchers who are trying to develop models of Internet structure and growth to have accurate and complete maps of Internet topology. Creating these maps is a non-trivial task as the the majority of network operators are commercial organisations and do not make public the structural details of their networks in a detail required for good science. Due to this, tools have been created which utilise features of the Internet Protocol (IP) to infer what Internet topologies look like. The technical details of these tools will be 1

10 src dst (a) direct path from src to dest src A B C D dst (b) realistic path from src to dest Figure 1.1: Example of a common misconception of Internet paths. A, B, C and D are interfaces of routers on the path between a source and destination. described in the next section. Due to the scale of the Internet, inferring topologies is both time consuming and network intensive. In order to improve the efficiency of topology discovery, a technique, doubletree [9], has been developed which significantly reduces the amount of network traffic and time required. However, doubletree has never been implemented within a system which is, or even can be, used by the general research community. Another problem with the conventional tools for Internet topology discovery is that they are prone to missing a significant amount of topology due to their assumption that there is only one path between two points in the Internet. There has been research into methods for discovering all of the possible paths[2]. However, the technique is even more time and resource intensive than the conventional methods. The goal of this work is to improve macroscopic Internet topology discovery to discover all possible paths without taking a prohibitively large amount of time and probes. 1.2 Internet Topology Basics Internet Topology Structure Most end users of the Internet do not know, and in fact do not need to know, the technical details behind how data gets from their computer to the host at the other end of the connection. In fact, to the end user, it could appear that their data travels directly to its destination as shown in Figure 1.1a. In 2

11 reality however, the data traverses many intermediate routers (hops), so the path between two points could look more like Figure 1.1b. The goal of topology discoveryistocorrectlyinferthepathbetweenanytwogivenpointsandthusthe setoflinkswhich makeup the path. Byselectingalargenumberofdestinations, and using a number of sources as vantage points, we can begin to assemble a map of the Internet Traceroute Because there is no easy way to discover Internet topology, a technique is used which exploits a feature of the Internet Protocol. The technique is known as Traceroute [11] and works by setting the Time To Live (TTL) field in the IP header of a packet which it then sends towards the destination of the path to be inferred. The TTL field is supposed to be used to prevent a packet from never exiting a routing loop. In most operating systems, the TTL field in the IP header is set to 64 when the packet is sent. Each router that the packet visits on the path, decrements the field by one. Once the value reachesone, the packet expires; the router at which it expires sends an Internet Control Measurement Protocol (ICMP) Time Exceeded packet back to the source telling the source that the packet did not make it to the destination. Traceroute uses the TTL field to elicit a response from a router at a defined distance into the network. The response contains the address of an interface at the router at which the packet expired. To build up a view of the path between a source and a given destination, traceroute starts by sending a probe packetwith a TTL of one, recordsthe addressof the responding routerand then sends a probe with a TTL of two and so on until a response is received from the address that it was probing toward. There are also other conditions which traceroute stops on depending on the implementation. For example, if a routing loop is detected or a series of hops are encountered which do not respond to probes. Because traceroute needs to send at least one probe per hop, in order to discover a complete path, many probes must be sent into the network. A 3

12 src A B D F dst (a) single path between source and destination B D src A F dst C E (b) two paths between source and destination Figure 1.2: Simple example of Load Balancing. Load balancing is used to distribute traffic over redundant links. path is comprised of a set of links which are inferred network links between the interfaces of two hops which have been discovered. An interface is an IP address of a router in the path. A router will have several IP addresses Load Balancing Traceroute assumes that there is only one path between any two given points in the Internet. In reality it is possible for there to be several different paths between a source and destination. An example of what paths can actually look like can be seen in Figure 1.2. This is possible due to routers which perform a function known as load balancing. Load balancing routers have at least two links which they can forward outgoing packets over thus increasing capacity Load Balancer Traceroute To be confident that all of the paths between a sourceand destination have been discovered given load balancing, a technique known as load balancer traceroute [3] is used. Load balancer traceroute works on the assumption that the majority of load balancing in the Internet is per flow. This allows load balancer traceroute to manipulate the paths that specific packets take through the network by manipulating the flow identifier of the packet. To discover a path with load balancer traceroute, a series of probes with varying flow identifier are sent to a hop in order to discover all the interfaces present. The reason that the 4

13 n 95% 99% n 95% 99% Table 1.1: Number of probes which must be sent to rule out a hop having n interfaces, given a confidence level of 95% or 99%. [3] flow identifier is varied is that if there is a load balancing router before the hop being probed, it will forward the packets out different links and thus different reply interfaces will be seen. Each hop is probed a certain number of times depending on the confidence level required and the number of interfaces discovered. Table 1.1 shows the number of probes that must be sent to have 95% or 99% confidence that all interfaces have been seen. Load balancer traceroute is discussed further in a later chapter. 1.3 The Problem The problem with using conventional traceroute and load balancer traceroute for large scale Internet topology measurement projects is that they are designed to be used for tracing from a single source to a single destination. When traceroute is used to probe a large destination list, it becomes expensive in terms of both network traffic and time. Also, the traffic generated by traceroute is unsolicited and so can appear to system administrators as port scanning or other malicious activity. Generally speaking, macroscopic topology mapping projects have no confidence that they have seen all of the topology and the technique to enumerate the complete topology is prohibitively expensive in both time and probes. 5

14 1.4 Contributions This project has made several contributions to the field of macroscopic Internet topology discovery. The first is the implementation and testing of doubletree, an existing efficient Internet topology discovery algorithm. Second a new algorithm, doubletree load balancer, which aims give confidence that a complete topology has been discovered whilst conserving time and network resources, has been proposed, implemented and tested. This report is structured as follows: this first chapter introduces the motivation of the work and the basics of Internet topology discovery upon which the rest of the report builds on. The second chapter provides some background into the field of macroscopic Internet topology discovery, putting this project in context. The third chapter introduces and describes the doubletree load balancer algorithm which this project has developed. The fourth chapter documents the implementation work carried out with regard to doubletree, doubletree load balancer, a distributed measurement control server and alias resolution. The fifth chapter reports on the results of testing the various components developed by this project. The sixth chapter concludes the report and describes future work that may be carried out. 6

15 Chapter 2 Background Because of the exponential growth of the Internet in the 1990s, monitoring its topology became an increasingly difficult task. Out of this dilemma were born several projects and techniques which aimed to create topological maps of the Internet. 2.1 BGP Border Gateway Protocol (BGP) is the de-facto external Internet routing protocol which allows routers to advertise the locations that they can forward packets to and the sequence of networks through which the packets will be forwarded. A BGP routing table consists of a list of prefixes, which are IP addresses ranges, that a router can forward packets to and the sequence of Internet Service Providers that the packets will be forwarded through. 2.2 Existing Topology Discovery Systems There are several topology discovery systems that have been used over the course of the last decade. They all differ slightly in the method that they use but their purpose is largely the same. That is, to collect data about Internet topology which can be used by researchers to create models of the Internet. Several of 7

16 these projects are based on having a set of geographically distributed vantages points which use some variation of traceroute to collect data skitter skitter [13] was one of the first large-scale projects to make use of a system of distributed monitors which constantly probe lists of addresses. skitter ran for ten years until February 2008 and had 18 monitors strategically positioned world-wide, many of which probed more than half a million addresses. skitter used traceroute to gather data about the routing structures of the Internet Ark Ark [4] is the successor to the skitter infrastructure. Ark has two primary goals. The first is to provide a platform which reduces the effort which is required to develop and deploy large scale measurement projects. The second goal is to enable the research community to run measurement tasks on a secure, distributed platform. Ark is a network of 27 monitors distributed throughout the world which all work in a coordinated fashion to conduct large scale measurements. Currently Ark is being used to perform large scale traceroute based topology measurements using team probing. Team probing means that the work is dynamically divided up between team members which allows the measurements to be made relatively quickly-between48to 56hoursforateam of13monitorstoprobe sevenmillion address ranges [4] Mercator Mercator [12] is another program which carried out large-scale mapping of the Internet. Unlike skitter, Mercator does not start with a list of known addresses to probe. Mercator uses a technique known as informed random address probing which allows the program to start with no state and generate addresses to 8

17 F src routed path E G src A B C D dest BGP path Figure 2.1: Source routing example. By using source routing, the default path for a packet can be overridden and the packet detoured through E, F and G. probe based on the subnet of the monitor. Mercator then branches out semirandomly, eventually probing almost all of the advertised, routable Internet Protocol version 4 (IPv4) address space. Rather than using multiple physical monitors spread around the world as skitter does, Mercator uses source-routing to bypass the standard BGP defined route to the destination. This allows Mercator to see paths through the Internet that a monitor probing from one static location normally would not. Figure 2.1 illustrates how by using source routing a packet can be detoured from its default route through the network. Only a small fraction of the routers in the Internet support source-routing (around 8%). However, every router that does support routing gives the same perspective as a Mercator instance running at that vantage point in the Internet. Because Mercator is designed to be fully self-contained, it has the advantage of not needing to rely on access to any external resources, such as BGP routing tables. There is a disadvantage to this feature however. This is that due to the semi- blind nature of Mercator s probing, far more probes are sent than are needed to form a map of the network. This, like skitter, means that large-scale, long-term probing of the Internet with Mercator could become intrusive and and an annoyance. 9

18 2.2.4 Rocketfuel Rocketfuel [20] is an attempt to carry out Internet Service Provider (ISP) mapping using an a method other than the brute-force, trace every address, method of the previous two systems described. Rocketfuel makes use of several techniques to carry out its mapping. The first technique is known as directed probing. Directed probing makes use of BGP routing tables to identify paths which traverse the ISP being mapped. Once the set of paths which transit the ISP is identified, Rocketfuel uses a series of path reduction methods to identify traceroute probes which probably take the same path within the ISP. At this point traceroutes are made using the probes identified. Once the traceroutes have been carried out, Rocketfuel uses a form of alias reduction to merge the different interfaces of a router into one. Routers have more than one interface and each interface has its own unique address. As traceroute only identifies interfaces, alias reduction is used to identify individual routers in the topology. Alias reduction (or resolution) is the process by which a router s aliases are identified and merged into a single representation of the router. Alias reduction methods are discussed in a later chapter. Rocketfuel is very efficient and performs the least number of traces possible to reliably map the links in an ISP. Because of this feature, it is rather unobtrusive and places very little load on the network infrastructure Summary While these projects have provided invaluable data to the research community, their continued viability is questionable due to the inefficient methods that are used. As the projects are scaled up, so is the amount of unsolicited network traffic that they create. A caveat of the brute-force approach that many of these systems use is that there is a set of links which will be probed many times due to the tree-like structure of the Internet. That is, the path from a vantage point into the socalled core of the Internet will be fairly consistent from trace to trace. This will 10

19 dst1 src1 B A src A dst2 src2 C dst C dst3 src3 B dst4 src4 (a) monitor rooted tree (b) destination rooted tree Figure 2.2: Internet path trees. (a) is an example of the tree which is seen when probing multiple destinations from a single vantage point. (b) is an example of the tree seen when probing a single destination from multiple vantage points. cause unnecessary duplication of effort by the monitor. Also, a facet of Internet routing design which causes several of these techniques to produce inaccurate results is load balancing. Due to load balancing, a different path can be seen with each probe sent into the network. This can result in the generated map not being true to the actual topology of the network. 2.3 Improving Traceroute Efficiency Doubletree Doubletree [9, 6, 7] is an algorithm which extends the classic traceroute algorithm to reduce the number of probes needed when tracing a large set of addresses from a distributed set of monitors. Doubletree exploits the tree-like properties of paths in the Internet. Internet paths form trees because of how the various commercial networks which comprise the Internet are connected. For a packet to reach any destination in the Internet, it must first traverse and exit the network of the ISP which is providing connectivity for the sender of the packet. The provider is likely to only have a couple of points which traffic can exit its network, it is at this point 11

20 that the trunk of the tree begins to branch. Each autonomous network that the packet travels through introduces further branching into the tree provided that it has multiple outgoing connections to other networks. The doubletree algorithm treats the paths between the monitors and the destinations as two trees as illustrated in Figure 2.2. One tree is monitor rooted, leading to the destinations which that monitor can reach. The other is destination rooted, leading to the monitors that can reach that destination. A monitor probes hop by hop along a path until it reaches an interface seen by other monitors probing that destination, assumes that it has reached a known branch and therefore the rest of the path to the destination is already known. Because there are two trees, there are two probing schemes in doubletree, forwards probing for the destination-rooted tree and backwards probing for the monitor-rooted tree. Doubletree makes use of these trees by remembering the interfaces from each tree that it has seen during probing. It does this by having two sets of data known as stop sets. There is the local stop set which is the a representation of the monitor-rooted tree, and the global stop set which is a representation of the destination-rooted tree. The local stop set contains all of the interfaces that a single monitor has seen while probing. The global stop set contains all the interfaces that the other monitors have seen probing a single destination. The global stop set is shared between all of the monitors participating in the probing; the local stop set never leaves the monitor. Doubletree starts probing at some number of hops h from the monitor. h is determined by using prior knowledge about path lengths from a monitor so that the value of h minimizes the probability that the destination will be reached with the first probe. This allows doubletree to start probing at a distance sufficiently far into the network to avoid re-probing hops close to the monitor, yet not so far that it is re-probing hops that other monitors have discovered while probing this destination. From h, doubletree probes forwards to h+1, h+2, etc., until it reaches an interface another monitor has seen before. It then starts probing backward from h-1 until it reaches an interface it has seen before. Because links close to the monitor or destination are only probed a small 12

21 number of times, rather than with almost every trace as in classic traceroute, doubletree is able to probe not only much faster than other large-scale algorithms, but it can also probe without causing the operators of routers and hosts to think that they are being targeted by a malicious denial of service attack. Until now, a problem with doubletree is that it has never been implemented into a system which is able to be used in the real world on a large scale. One of the contributions of this work is an integration of the doubletree algorithms into Internet topology mapping software (scamper) that is part of an existing project (Ark). 2.4 Improving Traceroute Accuracy When using conventional traceroute to probe hosts on the Internet, there are a number of factors which can cause the outcome of a trace to vary from the actual, real-world path taken through the network. In order to improve the accuracy of traces, these situations need to be taken into consideration and techniques employed to counter their effects Alias Resolution Routers have more than one interface connected to the Internet, as by definition, to forward packets, a router must be connected to more than one network. Each interface has an IP address assigned to it. This causes problems when discovering paths with traceroute, because traceroute only discovers interfaces. In other words, maps derived from traceroute data may contain more nodes than actually exist, as the mapping process assumes that each unique IP address represents a router in the path. Alias resolution is the process of folding the multiple IP addresses of a router into a single node in the map. Figure 2.3 shows how traceroute can infer a path which has more routers than actually exist, but once alias resolution is performed, interfaces can be assigned to the router topology. 13

22 E E src1 A C src1 A F C,D F src2 B D G src2 B G H H (a) before alias resolution (b) after alias resolution Figure 2.3: Alias resolution example. C and D are both interfaces on a single router. What appears to be two distinct networks before alias resolution turns out to be one interconnected network. x,y x1,y2 C E src A B x2,y1 G x2,y1,x1,y2 x1,x2,y1,y2 D F Figure 2.4: IPID usage. The packets identified by IPIDs x and y are fragmented by the router at A into x1, x2, y1 and y2. The load balancing router at B then forwards the fragments out links which have different speeds causing the fragmented packets to arrive at G out of order. The IPID allows the fragments to be uniquely identified and reassembled correctly into the original packets. Alias Resolution Methods There are several methods for performing alias resolution. One such method relies on routers using the IP address of the outgoing interface as the source address for ICMP port-unreachable messages [17]. This means that when two potential aliases are determined, packets can be sent to each of them, and if the source address of the replies are the same, then an alias has been found. This method is problematic as it assumes that there is only a single dominant route from all of the router s interfaces to a given destination. 14

23 Another alias resolution method (as used in Ally) [20], takes advantage of how the Internet Protocol Identifier (IPID) counter is implemented in routers. The IPIDfield is used in IP packetsto uniquely identify a packetfor re-assembly as illustrated in Figure 2.4. Most router implementations simply increment the IPID field for each packet they create. Using this knowledge, when a pair of potential alias addresses are identified, packets can be sent to both and by comparing the IPIDs of the packets received back from each. If the IPIDs received back are sequential (or close to each other) and in the correct order, it can be assumed that provided there is some variation in the sequence of reply addresses, the two IP addresses are in fact aliases for the same router. This method is fairly reliable, provided that routers increment the IPID counter in a sequential manner. Some vendors, the Linux kernel for example, set the IPID of all packets to zero. This makes it impossible to determine whether the addresses are aliases or simply routers which do not follow the convention of incrementing the IPID. One other method for resolving aliases is to perform some parsing of the Domain Name Service(DNS) names assigned to a router interface[19] to extract information about the router that they belong to. This method takes advantage of the tendency for ISPs to name their routers in a way which identifies where they are located geographically within the network. For example, sl-bb21-lon sprintlink.net and sl-bb21-lon-8-0.sprintlink.net are aliases for the same backbone router [19]. This technique has limited applicability in the doubletree system as the software needs to be trained for each ISP s naming system. Also, some ISPs do not use a systematic naming system, or simply do not even name their routers within the DNS. Another method suggested by Spring Te al. [19] is to use the maps generated from the traceroutes and two inference rules which allow alias addresses to be folded together. The first rule is that two adjacent addresses in the map are likely to represent adjacent routers rather than the same router. This is because if these addresses were in fact aliases, there would be a routing loop since one alias is forwarding to another alias on the same router. The other 15

24 rule specifies that IP addresses immediately before a point where links merge are likely to aliases if the links are point-to-point, as there would not be one address connected to two different routers with a point-to-point link. Project Applicability The inference rule method could be useful in the doubletree system; however it would need to implemented as a post-tracing script due to it needing a complete map to work. The advantage to this method over the Ally IPID method is that it requires no additional probing of the network. This is in keeping with our effort to reduce the load on the networks being probed. It would be fairly straightforward to implement the Ally IPID method in a doubletree system. Due to the nature of the processing performed to resolve aliases, it is not necessary for this process to be carried out at the same time as the actual probing and so could be implemented as an extra program which processes the output from the doubletree system. However, there may be some advantage to having aliases resolved in semi-real-time so that other monitors could take advantage of knowing other interfaces that a router has. This would allow probing to be stopped when a router visited previously is encountered rather than just interfaces previously seen. There has been some work done with regard to implementing Ally within the same framework as the other work that this report presents; this is also discussed later Paris Traceroute As links in the Internet become more plentiful, network administrators are employing load balancing techniques to enhance reliability and improve utilisation of the available infrastructure. There are three classes of load balancing. These are: per flow, per destination and per packet. Per flow load balancing works by attempting to forward packets which belong to a specific flow or connection out the same interface. A flow of packets is usually identified by a combination of the values in the IP header, specifically the source and destination ports 16

25 and addresses and the protocol. It has been found that routers may use any combination of these fields to determine a flow identifier [1]. Per destination load balancing is similar to per flow load balancing except that the only header field which is taken into consideration when forwarding packets is the destination address. Per packet load balancing is by far the least common method which works by simply sending packets over the first available link. The reason that per packet load balancing is so rare is because the different links likely have different bandwidths and so there is the very real risk that packets will arrive at the destination out of order. In terms of mapping, per-destination balancing is the same as classic routing and so does not need to be dealt with. Of the other two, the only (and indeed most common) situation that can be addressed in terms of traceroute, is perflow. In order to reliably probe past per-flow balancing routers, the traceroute packets must be manipulated in such a way that the router treats them all as being part of the same flow. This must be done in concert with the other requirement of a traceroute packet, that, its sequence in the trace can be identified once it is returned, so that the traceroute software can match the response with a probe that has been sent. As Augustin et al. [1] found, routers use a combination of the standard flow identifiers: source address, destination address, source port, destination port and protocol. They also found that routers can also use the IP Type Of Service (TOS), ICMP code and checksum fields to identify flows. This causes a problem for traceroute as traceroute traditionally varied the destination port field so that it could match responses with probes. In order to overcome this problem, the authors found that traceroute needs to use a field that is in the first eight octets of the header (so that ICMP includes it in the return packet) but is not used for load balancing to uniquely identify probes. For User Datagram Protocol (UDP) probes, the checksum is manipulated. For ICMP probes, Paris traceroute varies the sequence field, as does standard traceroute, but it also varies the identifier field so as to keep the checksum constant. 17

26 Project Applicability These improvements to the classic traceroute algorithm can be easily made so that the doubletree algorithm makes use of Paris traceroute. In fact, it is even more vital for doubletree to make use of Paris traceroute as doubletree makes assumptions about the path being constant between traces to allow it to stop probing at a previously encountered node. If paris is not used, then load balancing can cause a different path to be seen with later probing. 2.5 Improving Traceroute Completeness As mentioned earlier, there can be several paths between two points in the Internet. Paris traceroute addresses this issue by ensuring that each trace only takes a single path to the destination. It has been found that there are a large number of load balanced links in the Internet [2] so this issue needs to be considered seriously. Using the Paris traceroute algorithm means that there is the potential for detail to be missed as the alternate paths through the network may never be encountered. To ensure that the maps collected are as accurate as possible, modifications and/or extensions need to be made to the doubletree algorithm to allow it to traverse all possible paths. There are two existing improvements in the area of discovering a more complete topology; these are load balancer traceroute and windowed doubletree Load Balancer Traceroute The authors of the Paris traceroute algorithm proposed an algorithm which extends Paris traceroute to find the complete set of load balanced paths between a source and destination to a specified confidence level [3]. The algorithm works by taking advantage of how per flow load balancing works. By varying the flow identifier of the probe packets being sent in a methodical way, it is possible to discover all of the interfaces which are a given distance into the network. It still probes hop by hop as with conventional traceroute, but at each hop, the 18

27 TTL 4 D E F G >61 probes TTL 3 B C >31 probes TTL 2 A 16 probes TTL 1 src 8 probes Figure 2.5: Cumulative probe totals at each TTL with Load Balancer Traceroute. The number of probes growsin proportion to the number ofload balanced paths towards the destination. algorithm probes for the next hop multiple times (according to the confidence level required). This discovers a set of interfaces to which the current hop is connected. Then, by keeping state about which path a particular flow takes, each of the new hops are probed and so on until the destination is reached. The major drawback to load balancer traceroute is that in order to discover all of the topology, it must send a large number of probes into the network. That is, even if there is only one interface at a given TTL, that is, no load balanced links at a hop, it still sends eight probes to be sure that there are no others to 99% confidence. Figure 2.5 shows the cumulative number of probes that need to be sent to be confident that there are no other interfaces which have not been discovered in a simple scenario. Each time there is more than one interface discovered at any hop, the number of probes that must be sent to rule out further interfaces increases; the total number of probes sent grows to a large number quickly, even on a short path. Due to the nature of how load balancer traceroute works, it is a non-trivial task to integrate it with the doubletree algorithm. Several issues must be addressed such as choosing an appropriate place to start probing from. These problems and potential solutions are presented in the next chapter. 19

28 2.5.2 Windowed Doubletree The authors of the original doubletree algorithm have also done some work in improving the algorithm so that it is able to map a more complete topology [8]. The windowed doubletree algorithm assumes that most of path diversity occurs in the middle of the Internet and so the algorithm randomly varies the starting hop along the path within a predetermined window. Rather than beginning probing at a constant hop h, each monitor randomly picks a value for h within a window which represents the area in which most of the missing links are located. The authors claim that this simple change to doubletree discovers up to 16% more links than standard doubletree in their simulations. It should be noted however that this technique has never been tested outside of simulation. Also, the simulation data that it was tested on was collected using non-paris traceroute techniques which infer incorrect topologies. Although windowed doubletree makes a significant improvement to the coverage of doubletree for no extra cost, it still does not infer all possible links in a systematic and reliable way. It is still a best-effort approach which cannot provide a statistical confidence that all the topology has been seen. 2.6 Summary While there are a plethora of existing techniques which can be used for Internet topology discovery, they each have their own short comings; whether that be only discovering a subset of the actual topology or discovering the complete topology in an inefficient manner. To address this issue, we present in the next chapter, a new technique, doubletree load balancer which aims to give confidence that the complete topology has been discovered whilst probing in an intelligent and economical manner. 20

29 Chapter 3 Doubletree Load Balancer Traceroute Because the load-balancer traceroute algorithm in its original state is inefficient both in terms of run time and network load, there is a need for an alternative to be developed which reliably discovers all of the topology whilst being more efficient. One such algorithm has been designed and tested and is outlined in this chapter. 3.1 Overview From a high level perspective, the doubletree load balancer algorithm provides a way to determine the best TTL at which to start the standard load balancer algorithm so that it does not unnecessarily visit links that it has seen before. It also provides a way to stop at the earliest point from which the rest of the path has been seen before. In order to reliably determine a suitable starting point for probing, the doubletree load balancer algorithm begins by probing backward from a predetermined TTL, maintaining a constant flow identifier. This differs from doubletree which begins by probing forwards. Another point of difference from doubletree is that doubletree load balancer adds to the local stop set when probing forwards 21

30 rather than backward. To begin with, doubletree load balancer sends a probe to the firsthop TTL which is provided as a parameter to the algorithm. This firsthop TTL must be selected carefully to ensure that probing is not started either too early in the pathortoolate. Ifprobingisstartedtooearly, thenlargenumbersofprobeswill be wasted rediscovering topology. Conversely, if the firsthop is set too high then there will be probes wasted to probe back to a known point. Once a response has been received from the first probe, the interface which sent the response is checked against the local stop set. If the interface is not in the local stop set, then the firsthop value is decremented by one and another probe is sent. Decrementing the firsthop by one effectively finds the hop in the path prior to the one just discovered. When the reply to this new probe is received, it is again checked against the local stop set. If it is not contained in the local stop set then the firsthop is again decremented by one. This process is continued until either an interface which is in the local stop set is found, or probing reaches the beginning of the path. If the backwards probing happens to get all the way back to beginning of the path without finding an interface which is in the local stop set, then the load balancer algorithm is used in its original form. If an interface which is in the local stop set is encountered, the doubletree load balancer algorithm then sends another probe to the TTL at which the known interface was seen. The flow identifier for this probe is varied from the first to try and discover other interfaces which exist at this TTL. Several of these probes with varying flow identifiers are sent until the required number of replies have been received to be confident that all of the interfaces have been discovered at this TTL. This process is much the same as for the standard load balancer algorithm; as more interfaces are discovered, more probes must be sent. If there are no alternate interfaces discovered once the required number of probes has been sent, then the load balancer algorithm is started from that TTL. If there are other interfaces seen, doubletree load balancer moves into a convergence phase where it attempts to follow the paths from each of the interfaces discovered, backward until they converge to a single point. This 22

31 TTL TTL TTL TTL TTL 1 2 2,3 3,4 4,5 src A C D E F initial TTL=3 B Figure 3.1: Asymmetric load balancing causing false links (shown as dashed lines) to be inferred by doubletree. If probing is started at TTL 3, there are a number of links that doubletree would infer which do not actually exist. convergence point is where the load balancer algorithm is started from. To find the convergence point, doubletree load balancer probes backward fromeachoftheinterfacesdiscoveredinmuchthesamefashionthatitdidtofind the initial interface. The path from each interface is traced back systematically until all of the paths converge to a hop with a single interface. This allows confidence that any flow identifier used to probe with will cause probe packets to traverse this interface. Once the load balancer algorithm has been started, it runs as described in an earlier chapter. The only differences being that as it probes forwards, it adds the interfaces that it discovers to both the local stop set and the global stop set. It also checks each interface discovered to see if it is in the global stop set, if it is, then probing is halted as it is assumed that the rest of the path has been seen with previous probing, just like doubletree. 3.2 Challenges Load balancing introduces several challenges which all have to be addressed so that the doubletree load balancer algorithm can function reliably and produce valid data. One issue that has to be addressed is that when doubletree starts probing forwards midway into the path, it has no knowledge of the paths which come before that point. Therefore if there was asymmetric load balancing earlier 23

32 LSS: A, B, C start TTL A B C E G D F discovered interfaces: C, E, G missed interfaces: D, F Figure 3.2: Alternate path causing topology to be missed due to bad choice of start point. By probing forward using flow identifiers that reach C, the alternate path comprised of D and F is missed. in the path, probing forward could cause invalid topology to be inferred as it would not be known which flow identifiers traverse which parts of the path. This problem is illustrated in Figure 3.1. If doubletree load balancer was to start probing at TTL 3, it would see both C and D and then at TTL 4 it would see both D and E. From these replies, doubletree would incorrectly infer the topology shown by the dashed lines in Figure 3.1. This is why doubletree load balancer begins by probing backwards using a single flow identifier. Probing backwards with a constant flow identifier allows us to have confidence that the links that are being inferred are valid. Another challenge is determining how to decide on the start TTL for the load balancer algorithm. If we were to simply use the first interface that it encountered which was in the local stop set as the start point, then it is possible that there are alternate, parallel paths which have not been seen with previous probing. These alternate paths would either never be discovered or would in fact cause invalid links to be inferred, similarly to the example illustrated in Figure 3.1. Figure 3.2 shows an example where a parallel path causes topology to be missed. By probing the TTL where the first local stop set interface is found multiple times and then converging all the discovered interfaces, we can be confident that there are no other unseen alternate paths. One important modification that had to be made to the doubletree algorithm was when to add to the local stop set. Doubletree adds to the local stop set 24

33 only while it is probing backward. The reason that this is relevant to doubletree load balancer is that if doubletree load balancer was to add to the local stop set whilst doing its initial backward probing then it could not be assumed that given an interface in the local stop set, the path from the monitor to it is known. It is this assumption is what allows doubletree load balancer to start by probing backwards rather than forwards. Because doubletree load balancer cannot add to the local stop set whilst probing backward, it must add therefore add to it whilst probing forwards. The reason for doubletree not adding to the local stop set whilst probing forwards is uncertain, as the authors have stated that the local stop set consists of all interfaces already seen by that monitor [8]. In fact, the lead author of doubletree cannot remember why they chose to only add to the local stop set whilst conducting backward probing [5]. The reason that it is safe for doubletree load balancer to add to the local stop set whilst probing forwards (although there is no evidence to support why doubletree does not do this also) is that when there is no state held by a monitor, that is, it is the first trace being carried out, doubletree load balancer simply sets the load balancer algorithm going from TTL 1, recording interfaces into the stop sets. Therefore there is no risk that the local stop set will be populated with interfaces to which a path from the monitor has not been seen. It is important to note that even though the global stop set and the local stop set are both populated with the same addresses at the same time, they are still different and cannotbe combinedinto one. The localstop set is aset oforigin, interfacepairs, where the origin is always the same (the address of the monitor). This set is consulted with every new interface discovered to determine whether tracing can be stopped (a stopping condition). This look-up in the local stop set is made for all destinations. The global stop set is a set of interface, destination pairs where the destination is the address being traced when the associated interface is discovered. Only a subset of the global stop set is consulted for a stopping condition, that is, the set of entries where the destination address matching the address being probed. The global stop set is also shared among monitors. Per packet load balancers are a problem both for load balancing traceroute 25

34 per packet load balancer C A B E F start TTL G D possible links inferred: F >G, F >F Figure 3.3: Per packet load balancer causing invalid topology to be inferred. Because the load balancing at B is not based on the flow identifier, packets will arrive at F at both TTL 4 and 5 in an unpredictable fashion causing invalid links to be inferred. B C D E Figure 3.4: A per-packet load balancer (B) in a diamond configuration such that the per-packet load balancer does not have any effect on the links inferred. All possibly inferred links between B and E are valid. and doubletree. As their behaviour can not be influenced, they have the potential to cause doubletree load balancer to infer invalid topology. As Figure 3.3 illustrates, if doubletree load balancer settles on TTL 5 (F) as its start point and begins probing forwards, because of the per packet load balancer at B and the asymmetric paths afterward, doubletree load balancer can again infer invalid links. In this case, F is seen at both TTL 5 and 6. This is because a flow identifier which cause packets to be routed from B to D to F causes F to be seen at a distance of five hops into the path, whereas if the flow identifier causes packets to be routed from B to C to E and then to F, F will be at a distance of six hops into the path. In order to minimize the possibility of this occurring, once we have determined a start TTL, it then sends multiple probes with the same flow identifier to that TTL. If there is more than one interface seen in the replies, we infer that there is a potential per packet load balancer earlier in the path and probing is halted for that destination as there is the 26