Relay Path Selection for Peer-to-peer Voice-overIP Systems. Quang Duc Bui MSc., B.Eng.

Transcription

1 Relay Path Selection for Peer-to-peer Voice-overIP Systems A thesis submitted in fulfillment of the requirements for the degree of Doctor of Philosophy Quang Duc Bui MSc., B.Eng. Electrical and Computer Engineering College of Science, Engineering and Health RMIT University August 2010

2 Copyright by Quang D. Bui 2010 All Rights Reserved ii

3 To my wife, Duong, and my family. iii

4 ABSTRACT Selecting one or multiple peers to relay voice calls is a critical component of largescale peer-to-peer (P2P) voice-over-ip (VoIP) systems, such as Skype. The challenge is to determine good relay paths for a given voice call in a practical manner. We study different VoIP relay path selection schemes which are usually instantiated during the call session initiation and refreshed periodically. This not only allows end hosts behind Network Address Translations (NATs) or firewalls to establish voice calls but also allows communication in periods of poor performance to be switched to alternate relay paths quickly. We propose an enhanced version of existing algorithms for selecting VoIP relay paths and demonstrate that it improves voice quality. Then, we provide a series of simulations and analytical discussions on the correspondences of relay path performance to overlay network scenarios. We show that there are more opportunities for P2P VoIP systems to obtain good relay paths when selecting relay nodes located at highly connected Transit domains. For better relay path performance, we recommend to select relay nodes whose distances are less than four hops from the source. We also show that, in general, increasing relay node density can reduce relay path length but generate more hop overlaps to the default path due to the breadth first search manner in existing algorithms. Finally, we fully address the problem of selecting VoIP relay path by taking a new approach, including the development of overall formulation, a new network model to effectively identify the optimal solution space, and a new heuristic algorithm for selecting VoIP relay paths. iv

5 ACKNOWLEDGEMENTS I acknowledge with great pleasure the contributions of my supervisor, Professor Andrew Jennings, who accepted me to study with his team a few years ago. Without him this work would not have been possible. At the beginning of my study, I was not really imagining what kind of experience I was running in. Professor Jennings has provided guidance, knowledge, advice, and direction whenever it was needed. He has always been very patient and encouraging me whenever I met obstacles. I appreciated very much Jennings s very pragmatic approach to problem solving. I would also like to thank him for improving my writing skills, providing supports to me to participate in conferences. Besides that, a large part of the ideas presented in this thesis belong to Jennings. The simulations in my thesis work were based on the simulation platform that I was studied during the time working for the Traffic Engineering for Quality of Service in the Internet, at Large Scale (TEQUILA) project as a part of my Master program in University of Surrey, UK. Under this project, I have received guidance from Professor George Pavlou and Doctor Panos Trimintzios. I have learnt from them how to do research in the field of Internet Traffic Engineering, as well as to develop programs from the TEQULA s simulation platform. I would thank to Doctor Himanshu Agrawal in the School of Electrical and Computer Engineering, RMIT University, AU who was studying with me under the supervision of Professor Jennings. He has always been willing to spend his valuable time to discuss the problems with me with patience, and shared his experiences and expertise without any reservation. I would also thank to the research people working in the iplane project in the Department of Computer Science, Washington University, USA. During the time I was trying to make a simulation of the real Internet configuration, they have been very helpful to instruct me to use their inter-domain path latency database. All of my questions to them have been responded usefully and without delay. v

6 Finally, I would thank all of my family for their supports and encouragements during my study. I would not have done this without your supports. vi

7 TABLE OF CONTENTS ABSTRACT...iv ACKNOWLEDGEMENTS... v LIST OF PAPERS... x LIST OF TABLES...xi LIST OF FIGURES...xii CHAPTER 1 INTRODUCTION Problem Statement and Research Objectives Limitations of the Dissertation Contributions Dissertation Organisation... 5 CHAPTER 2 BACKGROUND Unstructured P2P Networks Non-hierarchical P2P Networks k-walker Random Walk Directed BFS and Routing Indices Based Search Local Indices Based Search Hierarchical Unstructured P2P Networks Summary Structured P2P Networks Non-hierarchical Structured P2P Networks Chord Pastry Hierarchical Structured P2P Networks Summary Locality-aware Peer-to-Peer Algorithms Network Proximity in Distributed Hash Tables Geographic Layout vii

8 Proximity Routing Proximity Neighbour Selection equus: a Locality-aware P2P System Summary Peer-to-Peer VoIP Systems VoIP Performance Parameters Skype Skype System Overview Skype Functions Skype Related Research P2P SIP Telephony Summary Relay Path Selection Resilient Overlay Network Detour PDF AS-Aware Peer-relay Protocol Earliest Divergence Rule Summary CHAPTER 3 AS-LEVEL TOPOLOGY AND SIMULATION DESIGN Introduction Related Work AS-level Topology The Design Requirements of Simulation and Analysis Tools Scale-Free, Hierarchical and ToR Networks Scale-Free Model and Network Hierarchy ToR Graphs AS Graph Generator - A Simulation Package The Real Internet Configuration Summary CHAPTER 4 MAXIMAL OVERLAP CLOSE RELAY Introduction viii

9 4.2 Related Work Minimal Overlap Close Relay Algorithm Comparison between Relay Path Selection Schemes Simulations and Analyses of Alternate Relay Path Selection Schemes Overlap in First Hop Relay Statistic Experiment with Different Relay Node Allocation Experiments and Analysis Using the Real Network Configuration The Optimal Hop-count Distance for Relay Nodes The Relay Node Population Summary CHAPTER 5 K-VIRTUAL SHORTEST RELAY PATH SELECTION Introduction Related Work Problem Statement, Modelling and Formulation Problem Statement Mathematical Formulation Network Model and Optimal Solution Space K Virtual Shortest Paths Algorithm The Algorithm The Costs of Using k-vsp Algorithm K-VSP Performance Evaluation Simulation Settings Performance Analysis Comparison between Two Versions of the k-vsp Algorithm Summary CHAPTER 6 CONCLUSION AND FUTURE WORK Summary of Contributions Future Work BIBLIOGRAPHY ix

10 LIST OF PAPERS Q. D. Bui and A. Jennings, "Relay path selection approaches in peer-to-peer VoIP systems," in Proc. of Australasian Telecommunication Networks and Application Conf. (ATNAC 08), 2008, pp Q. D. Bui and A. Jennings, "Relay node selection in large-scale VoIP overlay networks," in Proc. of 1st Intl. Conf. on Ubiquitous and Future Networks (ICUFN 09), H. Agrawal, A. Jennings, and Q. D. Bui, "A hybrid approach for Robust Traffic Engineering," in 4th Intl. Symp. on Wireless Pervasive Computing (ISWPC 09), 2009, pp x

11 LIST OF TABLES Table 2.1. Summary of different features in P2P applications [35] Table 2.2. The Mean Opinion Score for measuring call quality Table 3.1. Statistics of inferred AS relationships Table 3.2. AS node name notations Table 4.1. The main operations of algorithms Table 5.1. Summary of Performance Trends of Relay Paths xi

12 LIST OF FIGURES Figure 2.1. Centralised and decentralised (with or without hierarchical) P2P systems... 8 Figure 2.2. Classification of P2P networks... 9 Figure 2.3. Example Chord network [35] Figure 2.4. An example network consisting of 5 cliques. Nodes that are close-by belong to the same clique, share the same ID, and are responsible for the same set of data items [54] Figure 2.5. Skype network. There are three main entities: supernodes, ordinary hosts, and the login server [58] Figure 2.6. Difference between SIP-using-P2P and P2P-over-SIP architectures [35] Figure 2.7. Architecture of the Detour virtual Internet [77] Figure 2.8. ASAP system structure [56] Figure 2.9. Illustration of Earliest Divergence Rule [9] Figure 3.1. Graph skeleton to make sure of node reachability Figure 3.2. Adding additional links by following power-law Figure 3.3. AS degree distribution of the real Internet and exampled simulated SGB graphs: RouteViews - the real Internet degree distribution [56] vs. SGB sim randomly simulated graphs Figure 3.4. CDF of low degree nodes in simulated graphs and the real Internet graph Figure 3.5. CDF of the two data sets of latency Figure 4.1. A comparison between the calculated delay and measured delay of relay paths Figure 4.2. MOCR pseudo code Figure 4.3. Relay path selection in different schemes Figure 4.4. Hop overlap comparison between EDR, ASAP and MOCR (with 95% confidence intervals) xii

13 Figure 4.5. Delay comparison between EDR, ASAP and MOCR (with 95% confidence intervals) Figure 4.6. Scatter plot of overlap in the first hop relay vs. the whole paths Figure 4.7. Overlap distribution of MOCR paths achieved in different relay node allocations Figure 4.8. Delay distribution of MOCR paths achieved in different relay node allocations Figure 4.9. Average delay vs. Hop count constraints (with 95% confidence intervals) Figure Average number of hop overlaps vs. Hop count constraints (with 95% confidence intervals) Figure Performance comparison of relay paths generated by different algorithms with and without 3 hop-count constraint Figure Average delay deterioration of 3 hop-count constraint relay paths under different relay pool Figure Average overlap improvement of 3 hop-count constraint relay paths under different relay pool Figure Average delay of paths under different relay node density Figure Average hop overlap of paths under different relay node density (with 95% confidence intervals) Figure 5.1. The representation of paths as arcs in 2D-Euclidean space; the solution spaces are colored in grey Figure 5.2. Example of a default path with 4 intersects with the v-sp Figure 5.3. An example of long default direct path from AS A to AS C; A relays traffic to C via the multi-homed AS B Figure 5.4. The k-virtual-shortest-path pseudo-code Figure 5.5. Average one-way delay comparison between relay paths selected by algorithms, the default and the v-sp paths (with 95% confidence intervals) 109 xiii

14 Figure 5.6. CDF of one-way delay of relay paths selected by algorithms, the default and the v-sp paths Figure 5.7. Session rank of one-way delays Figure 5.8. Average overlap comparison between relay paths selected by algorithms (with 95% confidence intervals) Figure 5.9. Percentage of overlap values compared between algorithms Figure Average overlaps of relay path selected in different default path lengths (with 95% confidence intervals) Figure Average delay of relay path selected in different default path lengths (with 95% confidence intervals) Figure Number of default paths in different default path lengths (with 95% confidence intervals) Figure The relationship between path distance and hop overlap parameters (with 95% confidence intervals) Figure Classifying relay paths within the solution space and the corresponding mean values of overlap (with 95% confidence intervals) Figure Identifying relay paths inside and outside the solution space and the corresponding mean values of path distance (with 95% confidence intervals) Figure CDF of delay comparison between the two k-vsp versions Figure CDF of number of overlaps comparison between the two k-vsp versions 118 xiv

15 Relay Path Selection for Peer-to-peer Voice-overIP Systems A thesis submitted in fulfillment of the requirements for the degree of Doctor of Philosophy Quang Duc Bui MSc., B.Eng. Electrical and Computer Engineering College of Science, Engineering and Health RMIT University August 2010

16 Copyright by Quang D. Bui 2010 All Rights Reserved ii

17 To my wife, Duong, and my family. iii

18 ABSTRACT Selecting one or multiple peers to relay voice calls is a critical component of largescale peer-to-peer (P2P) voice-over-ip (VoIP) systems, such as Skype. The challenge is to determine good relay paths for a given voice call in a practical manner. We study different VoIP relay path selection schemes which are usually instantiated during the call session initiation and refreshed periodically. This not only allows end hosts behind Network Address Translations (NATs) or firewalls to establish voice calls but also allows communication in periods of poor performance to be switched to alternate relay paths quickly. We propose an enhanced version of existing algorithms for selecting VoIP relay paths and demonstrate that it improves voice quality. Then, we provide a series of simulations and analytical discussions on the correspondences of relay path performance to overlay network scenarios. We show that there are more opportunities for P2P VoIP systems to obtain good relay paths when selecting relay nodes located at highly connected Transit domains. For better relay path performance, we recommend to select relay nodes whose distances are less than four hops from the source. We also show that, in general, increasing relay node density can reduce relay path length but generate more hop overlaps to the default path due to the breadth first search manner in existing algorithms. Finally, we fully address the problem of selecting VoIP relay path by taking a new approach, including the development of overall formulation, a new network model to effectively identify the optimal solution space, and a new heuristic algorithm for selecting VoIP relay paths. iv

19 ACKNOWLEDGEMENTS I acknowledge with great pleasure the contributions of my supervisor, Professor Andrew Jennings, who accepted me to study with his team a few years ago. Without him this work would not have been possible. At the beginning of my study, I was not really imagining what kind of experience I was running in. Professor Jennings has provided guidance, knowledge, advice, and direction whenever it was needed. He has always been very patient and encouraging me whenever I met obstacles. I appreciated very much Jennings s very pragmatic approach to problem solving. I would also like to thank him for improving my writing skills, providing supports to me to participate in conferences. Besides that, a large part of the ideas presented in this thesis belong to Jennings. The simulations in my thesis work were based on the simulation platform that I was studied during the time working for the Traffic Engineering for Quality of Service in the Internet, at Large Scale (TEQUILA) project as a part of my Master program in University of Surrey, UK. Under this project, I have received guidance from Professor George Pavlou and Doctor Panos Trimintzios. I have learnt from them how to do research in the field of Internet Traffic Engineering, as well as to develop programs from the TEQULA s simulation platform. I would thank to Doctor Himanshu Agrawal in the School of Electrical and Computer Engineering, RMIT University, AU who was studying with me under the supervision of Professor Jennings. He has always been willing to spend his valuable time to discuss the problems with me with patience, and shared his experiences and expertise without any reservation. I would also thank to the research people working in the iplane project in the Department of Computer Science, Washington University, USA. During the time I was trying to make a simulation of the real Internet configuration, they have been very helpful to instruct me to use their inter-domain path latency database. All of my questions to them have been responded usefully and without delay. v

20 Finally, I would thank all of my family for their supports and encouragements during my study. I would not have done this without your supports. vi

21 TABLE OF CONTENTS ABSTRACT...iv ACKNOWLEDGEMENTS... v LIST OF PAPERS... x LIST OF TABLES...xi LIST OF FIGURES...xii CHAPTER 1 INTRODUCTION Problem Statement and Research Objectives Limitations of the Dissertation Contributions Dissertation Organisation... 5 CHAPTER 2 BACKGROUND Unstructured P2P Networks Non-hierarchical P2P Networks k-walker Random Walk Directed BFS and Routing Indices Based Search Local Indices Based Search Hierarchical Unstructured P2P Networks Summary Structured P2P Networks Non-hierarchical Structured P2P Networks Chord Pastry Hierarchical Structured P2P Networks Summary Locality-aware Peer-to-Peer Algorithms Network Proximity in Distributed Hash Tables Geographic Layout vii

22 Proximity Routing Proximity Neighbour Selection equus: a Locality-aware P2P System Summary Peer-to-Peer VoIP Systems VoIP Performance Parameters Skype Skype System Overview Skype Functions Skype Related Research P2P SIP Telephony Summary Relay Path Selection Resilient Overlay Network Detour PDF AS-Aware Peer-relay Protocol Earliest Divergence Rule Summary CHAPTER 3 AS-LEVEL TOPOLOGY AND SIMULATION DESIGN Introduction Related Work AS-level Topology The Design Requirements of Simulation and Analysis Tools Scale-Free, Hierarchical and ToR Networks Scale-Free Model and Network Hierarchy ToR Graphs AS Graph Generator - A Simulation Package The Real Internet Configuration Summary CHAPTER 4 MAXIMAL OVERLAP CLOSE RELAY Introduction viii

23 4.2 Related Work Minimal Overlap Close Relay Algorithm Comparison between Relay Path Selection Schemes Simulations and Analyses of Alternate Relay Path Selection Schemes Overlap in First Hop Relay Statistic Experiment with Different Relay Node Allocation Experiments and Analysis Using the Real Network Configuration The Optimal Hop-count Distance for Relay Nodes The Relay Node Population Summary CHAPTER 5 K-VIRTUAL SHORTEST RELAY PATH SELECTION Introduction Related Work Problem Statement, Modelling and Formulation Problem Statement Mathematical Formulation Network Model and Optimal Solution Space K Virtual Shortest Paths Algorithm The Algorithm The Costs of Using k-vsp Algorithm K-VSP Performance Evaluation Simulation Settings Performance Analysis Comparison between Two Versions of the k-vsp Algorithm Summary CHAPTER 6 CONCLUSION AND FUTURE WORK Summary of Contributions Future Work BIBLIOGRAPHY ix

24 LIST OF PAPERS Q. D. Bui and A. Jennings, "Relay path selection approaches in peer-to-peer VoIP systems," in Proc. of Australasian Telecommunication Networks and Application Conf. (ATNAC 08), 2008, pp Q. D. Bui and A. Jennings, "Relay node selection in large-scale VoIP overlay networks," in Proc. of 1st Intl. Conf. on Ubiquitous and Future Networks (ICUFN 09), H. Agrawal, A. Jennings, and Q. D. Bui, "A hybrid approach for Robust Traffic Engineering," in 4th Intl. Symp. on Wireless Pervasive Computing (ISWPC 09), 2009, pp x

25 LIST OF TABLES Table 2.1. Summary of different features in P2P applications [35] Table 2.2. The Mean Opinion Score for measuring call quality Table 3.1. Statistics of inferred AS relationships Table 3.2. AS node name notations Table 4.1. The main operations of algorithms Table 5.1. Summary of Performance Trends of Relay Paths xi

26 LIST OF FIGURES Figure 2.1. Centralised and decentralised (with or without hierarchical) P2P systems... 8 Figure 2.2. Classification of P2P networks... 9 Figure 2.3. Example Chord network [35] Figure 2.4. An example network consisting of 5 cliques. Nodes that are close-by belong to the same clique, share the same ID, and are responsible for the same set of data items [54] Figure 2.5. Skype network. There are three main entities: supernodes, ordinary hosts, and the login server [58] Figure 2.6. Difference between SIP-using-P2P and P2P-over-SIP architectures [35] Figure 2.7. Architecture of the Detour virtual Internet [77] Figure 2.8. ASAP system structure [56] Figure 2.9. Illustration of Earliest Divergence Rule [9] Figure 3.1. Graph skeleton to make sure of node reachability Figure 3.2. Adding additional links by following power-law Figure 3.3. AS degree distribution of the real Internet and exampled simulated SGB graphs: RouteViews - the real Internet degree distribution [56] vs. SGB sim randomly simulated graphs Figure 3.4. CDF of low degree nodes in simulated graphs and the real Internet graph Figure 3.5. CDF of the two data sets of latency Figure 4.1. A comparison between the calculated delay and measured delay of relay paths Figure 4.2. MOCR pseudo code Figure 4.3. Relay path selection in different schemes Figure 4.4. Hop overlap comparison between EDR, ASAP and MOCR (with 95% confidence intervals) xii

27 Figure 4.5. Delay comparison between EDR, ASAP and MOCR (with 95% confidence intervals) Figure 4.6. Scatter plot of overlap in the first hop relay vs. the whole paths Figure 4.7. Overlap distribution of MOCR paths achieved in different relay node allocations Figure 4.8. Delay distribution of MOCR paths achieved in different relay node allocations Figure 4.9. Average delay vs. Hop count constraints (with 95% confidence intervals) Figure Average number of hop overlaps vs. Hop count constraints (with 95% confidence intervals) Figure Performance comparison of relay paths generated by different algorithms with and without 3 hop-count constraint Figure Average delay deterioration of 3 hop-count constraint relay paths under different relay pool Figure Average overlap improvement of 3 hop-count constraint relay paths under different relay pool Figure Average delay of paths under different relay node density Figure Average hop overlap of paths under different relay node density (with 95% confidence intervals) Figure 5.1. The representation of paths as arcs in 2D-Euclidean space; the solution spaces are colored in grey Figure 5.2. Example of a default path with 4 intersects with the v-sp Figure 5.3. An example of long default direct path from AS A to AS C; A relays traffic to C via the multi-homed AS B Figure 5.4. The k-virtual-shortest-path pseudo-code Figure 5.5. Average one-way delay comparison between relay paths selected by algorithms, the default and the v-sp paths (with 95% confidence intervals) 109 xiii

28 Figure 5.6. CDF of one-way delay of relay paths selected by algorithms, the default and the v-sp paths Figure 5.7. Session rank of one-way delays Figure 5.8. Average overlap comparison between relay paths selected by algorithms (with 95% confidence intervals) Figure 5.9. Percentage of overlap values compared between algorithms Figure Average overlaps of relay path selected in different default path lengths (with 95% confidence intervals) Figure Average delay of relay path selected in different default path lengths (with 95% confidence intervals) Figure Number of default paths in different default path lengths (with 95% confidence intervals) Figure The relationship between path distance and hop overlap parameters (with 95% confidence intervals) Figure Classifying relay paths within the solution space and the corresponding mean values of overlap (with 95% confidence intervals) Figure Identifying relay paths inside and outside the solution space and the corresponding mean values of path distance (with 95% confidence intervals) Figure CDF of delay comparison between the two k-vsp versions Figure CDF of number of overlaps comparison between the two k-vsp versions 118 xiv

29 CHAPTER 1 INTRODUCTION 1.1 Problem Statement and Research Objectives Providing Quality of Service (QoS) services for the Internet has become a critical demand to the Internet due to the evolution of real-time applications, such as voice-overip (VoIP), Video on Demand (VOD) and video/audio instant messaging. A large number of architectures, technologies, and mechanisms enabling IP QoS have been created to enhance the conventional IP service model. The most important frameworks and technologies proposed are Integrate Services (IntServ) [1], Resource reservation Protocol (RSVP) [2], Differentiated Services (DiffServ) [3], and Multi-Protocol Label Switching (MPLS) [4]. However, delivering end-to-end QoS in the Internet remains a challenging task. An important reason for the lack of deployment of many of these approaches is that the Internet has already become a social infrastructure comprised of a large number of independently operated networks, i.e. autonomous systems (ASs), where peering points provide the connection of separate ASs of the Internet into one cooperating infrastructure. It is difficult to widely deploy new approaches that significantly change the existing architecture of the network such as those proposed in IntServ or DiffServ. The economics of peering make the provisioning of end-to-end QoS unlikely. Whereas most peering agreements are bilateral contracts between ASs at peering points, end-to-end QoS is a cooperative effort of all ASs on an end-to-end path of a flow with service guarantees. Although an Internet Service Provider (ISP) may have an interest in providing QoS guarantees within its own AS, there is a lack of incentive to support similar service guarantees to customers of remote ASs. 1

30 To overcome these end-to-end QoS interoperability issues, peer-to-peer (P2P) overlay networks have been developed as a higher level mechanism that can support new services to users on top of the network layer without requiring changes to the infrastructure or its business practices. The key to the success of such overlay path switching, is the availability of diverse paths. Most participating nodes in P2P systems can function as traffic relaying points for other nodes. It realises possibilities of constructing a rich set of alternate relay paths which potentially help to bypass performance degradation. P2P implementation is also cost-effective for small companies and even individuals. Over a decade, P2P systems have attracted attention from Internet users as well as research community. From Napster and ICQ, the earliest P2P systems, emerging in 1999, more and more popular systems such as Gnutella, KaZaA, BitTorrent and Skype have been realised. Motivated by practical successes in VoIP deployment via P2P overlay networks, it is envisioned that overlay systems can be a promising alternative for end-to-end QoS delivery in IP networks. Decentralised structured peer-to-peer systems inherently have high scalability because the capacity scales with user population, robustness and fault tolerance because there is no centralised server and the network self-organises itself. The key to this approach is to find among a rich set of routing paths an alternative route that can avoid congestion, and hence, meet the QoS demands. The objective of this work is to explore new approaches to select alternate relay paths that are suitable for transmitting VoIP traffic. In P2P systems, there is a trade-off between performance and overhead (and hence, latency) in selecting good QoS candidates. This is particularly important for large-scale networks. Large-scale P2P networks are vastly decentralised, and the host population provides an enormous diversity of alternate relay paths. It is however a very dynamic system where peers frequently join and leave. Selecting relay paths by a brute-force solution that continuously monitors all possible alternatives is not feasible in this situation. On the contrary, randomly selecting a relay can often result in poor options. Existing 2

31 researches, e.g. [5], [6], [7], [8], [9], [10], suggest that ideal relay paths should exhibit good end-to-end performance characteristics with maximal diversity, i.e. how significant relay paths deviate from those of default path, because disjoint paths are unlikely to experience link degradation or failure at the same time. The first research question which needs to be carefully considered is how to quickly identify suitable QoS VoIP relay paths in a scalable way? Since P2P systems operate on top of physical network, their path selection is performed by end hosts running applications according to their QoS requirements, and the underlying IP routing infrastructure is not aware of overlay routing activity. In this sense, P2P routing is also regarded as selfish routing since it does not consider other traffic within the network. The selfish routing optimisation problem, in theory, has been addressed by using the non-cooperative multiple optimisation approach for game theory and economics which first proposed by J. F Nash [11]. A crucial property of selfish routing is that it is possible to reach a traffic equilibrium which is known as the Nash equilibrium. That is, traffic equilibrium is reached in such a way that all routes that are used between a source-destination node pair have equal costs while all unused routes have a higher cost [12]. The main problem of Nash equilibria is that it is difficult in practice to exhibit such an equilibrium [13] because of the complexity in designing such a protocol and frequency of overhead exchange between entities. Therefore, an interesting research question is how to quantify the optimisation problem of selecting VoIP relay paths given a set of optimal objectives? And hence, can we practically obtain near-optimal solutions for the problem in large-scale networks? 1.2 Limitations of the Dissertation In this dissertation, we investigate relay path selection algorithms proposed for large- scale P2P VoIP systems with regard to two significant performance metrics: path latency and path disjointness. While latency is, obviously, very sensitive to VoIP service, path 3

32 disjointness is considered to improve performance since disjoint backup relay paths are unlikely to experience link degradation or failure at the same time. We later justify our selection of these performance parameters in Section Due to the stringent time requirement of voice calls, the task of selecting suitable relay path accommodating voice traffic needs to be rapidly processed. It is expected that either the relay candidates are available before making a voice call or they should be selected within the period of call initiation (i.e. several seconds at most). These are the prime conditions when we consider relay path selection algorithms. P2P networks can be classified based on the control over data location. There are two main categories: unstructured and structured networks. Unstructured P2P systems use two kinds of searching, data look up and keyword searching. Structured P2P networks usually employ distributed hash tables (DHTs) which typically support data lookup functionality [14]. More discussion about this topic can be found in the later sections. This work assumes that P2P systems hold efficient search functionalities and protocols which allow peer operation and basic network management. Algorithms studied in this work provide advanced functionalities for localising and selecting relay peers in order to improve the search performance as well as the quality of consequent VoIP calls. Specifically, we study algorithms which associate topological information of large-scale networks to the task of selecting relay paths. We focus on AS-level path, as opposed to physical IP-level path, because this greatly reduces the amount of topology information that needs to be maintained by end systems. It has been argued by T. Fei et al. [8] that this is a more scalable solution for large P2P networks. 1.3 Contributions Our project has three major contributions. Firstly, we propose an enhanced version of existing algorithms focusing on techniques used to select cross-domain relay paths in the large scale VoIP systems. For scalability and efficiency, our heuristic employs AS-level path information so that topological information maintained at end hosts can be 4

33 significantly reduced while exhibiting better path performance. The proposed heuristic utilises the availability of large number of relay peers in large scale P2P systems and predetermines, possibly during the call instantiation or periodic update, a pool of close relay candidates in which VoIP traffic can be effectively switched to in the events of network failure or degradation. We illustrate that the proposed algorithm can bring better quality paths with the same constraints as existing methods. Secondly, we provide a series of simulations and analytical discussions on the correspondences of relay path performance to overlay network scenarios. We point out that there are more opportunities for P2P VoIP systems to obtain good relay paths when selecting relay nodes located at highly connected Transit domains. We then quantify the lower threshold for the first hop relay path length. It is recommended for achieving better relay path performance to select relay nodes whose distances are less than four hops from their sources. We also show that, in general, increasing relay node density can reduce relay path length but generate more hop overlaps to the default path due to the breadth first search of existing algorithms. Thirdly, we fully address the problem of selecting VoIP relay path by taking a new approach, including the development of overall formulation, a new network model, and a new heuristic algorithm for selecting VoIP relay paths. Our formulation and network model can be used to effectively present, identify the optimal solution space for the problem of VoIP relay path selection with regard to path latency and path diversity objectives, which has not been addressed before. Using this scheme of localising the optimal solution space, we develop a new algorithm for selecting alternate relay based on latency, hop overlap and the valley-free rule in the AS-level Internet. 1.4 Dissertation Organisation The remainder of this thesis is organised as follows. In Chapter 2, we motivate our research objective of using P2P to provide QoS for VoIP systems through discussion on the evolution of P2P systems, the benefit of applying P2P to VoIP networks. In this 5

34 chapter, we also provide an overview of the key technologies in P2P and a discussion on VoIP quality performance. In Chapter 3, we discuss the some important aspects in simulating AS-level Internet topology. Subsequently, we introduce two AS graph simulations. Chapter 4 explores methods for identifying suitable VoIP overlay paths in a scalable manner. We discuss and compare different path selection schemes in P2P overlay networks, and introduce a heuristic algorithm for alternate relay path selection. We then look at the choice of relay nodes and its impact on the performance of VoIP overlay paths in interdomain environment. Using simulation of the real Internet configuration, we quantify the lower threshold for the first hop relay path length in hop-count. In Chapter 5, we fully address the second research question by modelling, formulating the problem of optimal VoIP relay path selection with regard to path latency and path diversity objectives. We then describe a complete solution for selecting alternate relay paths in VoIP systems based on latency, hop overlaps and the valley-free nature in the current interdomain routing environment. Finally, Chapter 6 concludes the major results of this dissertation as well as suggesting future work. 6

35 CHAPTER 2 BACKGROUND Peer-to-peer (P2P) networks are overlay networks, where nodes are end hosts in unrelated administrative domains in the Internet. Nodes in a P2P system play equal roles; hence, they are also called peers. These peers form a virtual overlay network on top of the physical Internet links by maintaining information about a set of other peers (i.e. neighbours) in the P2P layer. Benefits offered by P2P networks include: (1) No special administration and financial engagements required; (2) Distributed and decentralised; thus, they are potentially fault tolerant and load balanced; (3) Self organised and adaptive. Central to P2P systems is the ability to efficiently lookup and locate data items and to manage them accordingly. Many aspects of P2P systems rest on this functionality. In contrast to centralised server applications, decentralised systems store content in multiple, possibly distant locations within the system. There are two approaches to provide the control over data location and network topology in P2P systems, unstructured and structured. In unstructured P2P networks, there is no rule that defines where a data item is stored and the network topology is symmetric. Searching functionality is typically provided through the flooding of query messages. In structured P2P networks, network configuration and the data locations are precisely defined [15]. Decentralised P2P networks can be further classified into non-hierarchical and hierarchical based on whether the arrangement of the P2P network is a hierarchy (Figure 2.1). In general, the hierarchy of a P2P system is exhibited by the roles that peers participate. In non-hierarchical (or pure P2P) systems, peers are totally equal in terms of the role on the network. In hierarchical P2P systems, such as hybrid systems, some peers serve as a super-peer for a set of normal nodes. Generally, pure P2P systems offer high 7

36 resilience and load balancing. However, they cannot take advantage of node heterogeneity, scalability and routing efficiency as the hierarchical P2P systems can [15]. Figure 2.2 provides a summary of P2P system classifications used in this dissertation. Note that a hybrid P2P (indicated with dash lines), in practice, may refer to either a hierarchical system (i.e. a combination of centralised and decentralised networks) or a combination of unstructured and structured P2P protocols such as Yappers [16]. P P S P P P P P S = server P S S P S = superpeer S P P P P P = peer P P = peer P P = peer Centralised P2P Nonhierarchical decentralised P2P Hierarchical decentralised P2P Figure 2.1. Centralised and decentralised (with or without hierarchical) P2P systems. This chapter provides the fundamentals of P2P routing and its application to voiceover-ip (VoIP) systems. Particularly, we focus on the searching techniques used in decentralised peer-to-peer systems which show our motivations. The remainder of this chapter is organised as follows. In Sections 2.1 and 2.2, we provide brief overviews of some basic searches in unstructured and structured P2P networks, and discuss their advantages and drawbacks. In Section 2.3, we investigate in more detail some advanced locality-aware searching schemes in structured P2P networks that motivate our research. Section 2.4 introduces popular P2P VoIP systems. Section 2.5 relates our work to recent relay path selection schemes. 8

37 Physical configuration Systems Centralised Decentralised Hybrid/Hierarchical Search functionality Unstructured Structured Hybrid Hierarchy Nonhierarchical Hierarchical Nonhierarchical Hierarchical Figure 2.2. Classification of P2P networks 2.1 Unstructured P2P Networks In this section, we describe the main features of unstructured P2P systems, using combination of information from [14], [15]. More detailed discussions on P2P networks can be found in the same book [17]. In both centralised and unstructured P2P systems, no rule strictly defines where data are stored and which nodes are neighbours to others. These systems either rely on lookups via a central server which stored the locations of all data items or use a flooding technique. In centralised P2P approach, e.g. Napster [18], P2P application at an end host needs first to lookup the location of a data item via a server, and then, the data are transferred directly between peers. Flooding based approach in unstructured P2P systems, such as Gnutella [19], sends lookup probes based on Breadth First Search (BFS) to all peers participating in the system with depth limit D, where D refers to the system-wide maximum number of hops of a query probe. Thus, flooding forms the maximum number of neighbours within a 9

38 ring centred at the source node with the radius of D-hops. It can be seen that the serverbased system suffers from exhibiting a single point of failure as well as being a bottleneck with regard to resources such as memory, processing power, and bandwidth. On the other hand, the flooding-based approach generates a large number of overheads, in which many are replicated, inducing a high consumption of network bandwidth. As a result, neither approach scales well. Alternative schemes have been proposed to address the problem of flooding, including BFS based or Depth First Search (DFS) based approaches. The BFS based scheme includes k-walker random walk [20], iterative deepening [21], directed BFS [21], intelligent search [22], local indices based search [21], adaptive probabilistic search [23], etc. The routing indices based search [24] and the attenuated bloom filter search [25] are variations of DFS. Their main purpose is to try to reduce overhead and to increase quality of query results. Searching schemes in unstructured P2P systems can also be classified as blind and informed searches; or deterministic and probabilistic searches. In an informed search, peers store some metadata to facilitate the search; while in a blind search, they do not keep information about data location. On the other hand, deterministic search means query forwarding is deterministic, while in probabilistic approach, query is probabilistically or randomly forwarded, or it is based on some kinds of ranking. In the next sub-sections, we investigate several unstructured P2P searching schemes Non-hierarchical P2P Networks k-walker Random Walk The random walk algorithm [20] further improves the search efficiency in the Gnutella system. In this scheme, the querying node forwards query message, i.e. called walker, to k randomly selected neighbours. These neighbours repeatedly choose k of their neighbours and forward the walker to those neighbours. The procedure continues until the desired data item is found. Obviously when k = 1, the amount of message overhead is 10

39 reduced but it causes the longest searching delay. Increasing k reduces the routing delay by k times, on average, as compared to the case of single walker because there are k times more nodes reached on the same number of walks. The main issue for the k-walker random walk algorithm is data replication. The works in [20] and [26] have tried to address the problem, in terms of the average search overhead for completed queries, by analysing and simulating three replication strategies: uniform, proportional, and square-root. Their results show that the uniform and proportional replications achieve the same average search size and larger than one achieved by squareroot strategy. Square-root replication also has smaller utilisation rate then others. Thus, square-root replication can be practically implemented by replicating query data proportional to the number of sites probed Directed BFS and Routing Indices Based Search In directed BFS scheme [21], the source node only sends query messages to a set of its neighbours that meet a certain criteria. The most common criterion used is returning history of high-quality results. Consequently, the selected neighbours forward query in the same fashion of BFS. Forwarding the query to a subset of neighbours helps the algorithm reduce the amount of routing overhead. By selecting good neighbours, it is possible to maintain the quality of query responses and to lessen routing delay. Heuristics for selecting good neighbours include: The closest neighbour, identified by number of hop-counts in previous messages; The most stable neighbour, identified by highest number of returned queries; and The lowest load neighbour, identified by the message queue delay. However, the message replication in this algorithm is not greatly reduced because apart from the source, all downstream peers involved in the query procedure still broadcast the query based on BFS. To further reduce overhead, routing indices based search is proposed in [24]. It is similar to directed BFS in that all nodes use information about their neighbours to guide the search. However, routing indices uses intelligent neighbour selection for the entire 11

40 search process, not only the querying node as in the case of directed BFS. In this scheme, a routing index (RI), a distributed data structure, facilitates a node to choose the best neighbours to forward query messages. Given a content query, the algorithm computes the top several best neighbours based on this data structure. Since an RI indicates route to one of its neighbours, instead of destinations, the size of routing table is reduced. In practice, routing indices based search is proposed for content query, e.g. a request for documents that contain a particular word. Hence, a good neighbour is typically the one through which many documents can be quickly found Local Indices Based Search The basic idea of the local indices [21] is to get the same number of query results as flooding based with less number of nodes processing a query. Each node in a local indices network maintains metadata (i.e. indices) on all nodes within k-hop distance. Local indices network employs the policy P for all nodes which is typically a list of depths to reach to other nodes. Using P, each node can directly respond queries from nodes, whose depths are listed in P, by checking its local indices and returning the query results without moving the query to other nodes. However, if their depths are smaller than the depth limit defined in P, nodes continue to forward query messages to all other neighbours. On the other hand, if a node is not listed in P, it just simply forwards the query to all of its neighbours and does not check the local indices. When a node joins, leaves or modifies its local indices, all nodes within k-hop distance from it will update their local indices accordingly and broadcast an update message to all of their neighbours. Once receiving the update message, those neighbours check their local indices whether this update contains information that affects their metadata. In summary, the local indices based search uses broadcast of the query message based on a list of depths. Only a selected number of nodes which are stated in the policy P process the query. As a result, this technique is claimed to greatly reduce the aggregate 12

41 cost of processing queries over the entire system, while maintaining equally high quality of search results Hierarchical Unstructured P2P Networks To overcome the drawbacks of flooding, dynamic hierarchy has been introduced for unstructured P2P systems so that not every query message has to be flooded through the whole network. The hierarchical (or hybrid) P2P is launched by the use of a special type of relay peer called superpeer or supernode. A superpeer is a peer which typically has high capacity. All superpeers connect to each others as a pure P2P network. Each superpeer operates as a centralised server to a set of normal nodes called leaf-nodes. However, they should not have more than 50 to 100 leaf-nodes [27], depending on its processing power and the bandwidth connection, to maintain the advantage of self-organisation and decentralisation. Each ordinary node may connect to one or more superpeers. For each query, the querying node forwards query message to one of its superpeers and waits for query results from this superpeer. In hierarchical P2P networks, queries are processed by superpeers only. The mechanisms that superpeers are used include flooding or random walk. This organisation allows searches in hierarchical networks to be implemented more efficiently than in ordinary unstructured P2P systems as the amount of flooded messages is reduced significantly Summary Unstructured P2P networks are extremely resilient to node joining and leaving and adapt well to the changing peer population because there is no special structure needed to maintain and no control over data storage. Since their searching mechanisms are often based on flooding, they suffer from generating potentially huge amount of signalling traffic. Such system design is broadly argued not to be scalable despite the fact that many real systems are built based on this scheme. To restrict the network bandwidth consumption, popular unstructured P2P systems employ constraint algorithms which in 13

42 fact may try to terminate a data query prematurely before the desired data is found. For example, Gnutella uses flooding with limited time-to-live (TTL) for query messages. As a result, unstructured P2P cannot guarantee the quality and performance of resource discovery. 2.2 Structured P2P Networks Structured P2P networks are considered the second generation of P2P. In structured P2P networks, there is no central directory but tight control over the P2P network topology. The neighbour relationship between peers and data locations is strictly defined. There is close coupling between network topology and resource location data. Searching in structured P2P networks is determined based on the particular P2P topology, which should be reconstructed along with the join and leave of peers. Structured P2P networks can guarantee finding a resource (data item) within bounded hops. Most structured P2P systems implement a distributed hash table (DHT) which is an approach for distributed and content-addressable data storage. DHTs employ additional techniques to manage data structures, to add redundancy, and to locate the nearest instances of a requested data item. A DHT contains table entries that are distributed among different peers located in arbitrary locations. Each data item is hashed to a unique numeric key representing for a namespace. Each peer in the network is responsible for a certain number of keys, i.e. a small part of the namespace, and is assigned a unique peer identifier. The DHT search supports two basic operations, including lookup(key), and put(key). The lookup(k) operation is used to localise (e.g., to return the IP address of) a peer responsible for the key k; and, the put(k) operation is used to store a data object (or a pointer to the data object) with the key k in the peer responsible for k. A peer must publish data objects using put(k) operation that were originally stored on it before these objects can actually be retrieved by other peers [15]. To implement the lookup operation, each peer in a DHT network maintains a forwarding table that manages mappings between keys and information (e.g., IP address) 14

43 of a set of its neighbours. When a peer receives a lookup request, it checks whether it can answer the request itself; if it cannot, the request is forwarded to one of its neighbours based on the forwarding table. The process continues until the result, as long as it exists, is found. Structured P2P systems are usually designed to handle the strong dynamics of peers, to scale to large number of potentially faulty peers based on their highly structured DHT protocols Non-hierarchical Structured P2P Networks The most well-known DHT schemes are Chord [28], Pastry [29], Tapestry [30] and CAN [31]. These schemes are based on particular flat data structures representing some traditional interconnection topologies, including ring, mesh, hypercube and other more special graphs such as the d-torus topology (used in CAN), de Bruijn graph used in Koorde [32], Butterfly topology used in Viceroy [33], etc. The underlying topologies have significant impacts on the performance, resilience, and other properties of DHT schemes. We briefly present the two most popular systems: Chord and Pastry Chord Chord [28] uses a ring data structure. Chord s peer identifiers form a uni-dimensional, circular identifier space. Both peers and keys in a Chord system are assigned m-bit identifiers based on a public hash function, such as SHA-1 [34] that generates 160-bit identifiers. Keys are assigned to peers as follows. Key k is assigned to the peer (called the successor) whose identifier is equal or follows the identifier of key k in the identifier space. Each peer keeps a finger table, i.e. list of neighbours, with the size, at most, of m entries. A finger entry contains the identifier, the IP address of the relevant node and some additional routing information. The additional information is not essential for the lookup operation; however, it helps to accelerate the search process. The i th entry in the finger table at peer n contains the first peer s that succeeds n by at least 2 i, where 0 i m and all the arithmetic is modulo 2 m. Peer s is called the i th finger of peer n and is calculated with the following formula: 15

44 finger i successor n 2 i mod 2 m (2.1) When a peer receives a lookup request for a given key, it checks for the closet preceding peer in its finger table and forwards the key to that peer. This process is repeated recursively until the peer responsible for the key is found. In the example in Figure 2.3, when peer 10 wants to find key 32, it looks up the finger table to find the closest match as start value of 26, and sends the query to peer 26. Similarly, peer 26 in turn sends it to peer 30, which finally sends it to peer 33. Peer 33 is the successor peer for the identifier 32 in the network, hence is responsible for storing information about key 32. The lookup latency is O log N because, in each step, the query is forwarded to at least half the remaining distance around the ring [35]. Figure 2.3. Example Chord network [35] Pastry Pastry [29] uses a tree based data structure which can be generalised as a hypercube. The peer identifier is 128-bit in base 2 b, where b is practically 4. Each peer A maintains a set of leaf nodes L in which the identifiers of half of the nodes are closet to and smaller 16

45 than the identifier of peer A, and the identifiers of the remaining leaf nodes are closet to and larger than the peer A s identifier. This structure guarantees the correctness of routing because the delivery is guaranteed unless L / 2 peers with their adjacent peer identifiers fail simultaneously. In order to shorten the routing time, each Pastry peer also keeps a routing table to other peers in the identifier space. Each peer A keeps 2 b 1 entries for each prefix of its identifier. Each entry in row n refers to a peer whose identifier shares the first n digits with identifier of the current peer A, but whose n 1 digit has one of the th 2 b 1 remaining other possible values than the present n th digit of the current peer A. Given a query request for the key k, peer A first tries to find in its leaf set a peer whose identifier is numerically closet to key k and to forward the query to that peer. If such peer does not exist, peer A tries to find a peer in its routing table whose identifier shares a longer prefix with key k than A. If no such a peer, A forwards the query to a peer whose identifier has the same prefix to A but is numerically closer to key k than A. Pastry also uses additional proximity heuristics during the process of forwarding queries. To achieve the routing latency of O log N, each peer needs to maintain O log N routing state. The searching schemes used in Tapestry [30] and Kademlia [36] are variants of Pastry Hierarchical Structured P2P Networks With the same notion of hierarchical unstructured P2P systems, hierarchical DHT P2P networks organise peers into different clusters. Each cluster forms its own overlay which all together constitutes the entire hierarchical overlay topology. Peers in the higher tier of the network are called dominating peers or superpeers, which generally require more computing resource, network bandwidth and take more responsibility in routing than normal peers. In most self-organised P2P hierarchical systems, an ordinary peer can become a superpeer based on one or several criterion, which depend on design of the system for a particular application. These criterion comprise of CPU power and storage capacity as 17

46 proposed by Mizrak et al. in [37], or node stability, connection quality and network bandwidth as in Garces-Erice et al. [38]. The superpeer selection scheme is different for each system. Usually, if a superpeer detects a better peer within its sub-network, the superpeer can promote this peer as superpeer and demote itself to an ordinary node. In the case of superpeer failure, which can be detected through periodic keep-alive messages between peer neighbours, the P2P systems can use protocols such as volunteer service [37] to select another superpeer in order to take over the load of the failed one; or the affected peers may change over to existing neighbour superpeers. Searching operations in each tier of the hierarchy can be similar to one of the non-hierarchical structured P2P schemes mentioned in previous sections. Examples of hierarchical structured P2P systems include Kelips [39], Coral [40], Brocade [41], HIERAS [42], KaZaA [43], etc. More detail about operations of hierarchical structured P2P systems can be found in Section 2.3 when we discuss locality-aware P2P systems Summary Structured P2P networks have attracted the attention of the research community because of their advantages including scalability, self-organisation and generality. They have bounded guarantees for resource discovery in which any existing resource can be located within predetermined number of hops. These desirable characteristics of structured P2P networks are realised by the use of DHT schemes. They provide a locationindependent naming infrastructure for constructing different types of applications. In hierarchical structured P2P systems, search queries can be processed even more efficiently because superpeers act as centralised servers. These systems also enjoy the advantages of the decentralised topology, i.e. resilience, stability, and load balancing because there are relatively more superpeers in the systems to dismiss the potential problem of becoming a single point of failure. Structured P2P systems are considered costly for maintenance because of the complex operations of DHTs such as maintaining forwarding tables that should be updated 18

47 whenever peer join or leave. Furthermore, superpeers in hierarchical structured systems play an important role in the top of the hierarchy; hence, failure of some of them might have serious impact on the systems, e.g. Skype registry servers failure in 2007 [44]. Nonetheless, structured P2P networks and their related protocols, algorithms are still a very attractive research topic and are considered future evolution of P2P. Current trends of research include the development of flexible searches [45]; the selection of superpeers so that it is possible to maximise the efficiency, to speed up routing [46]; the correspondences between the overlay P2P networks to the underlying physical network [47], [48]; the relay path diversity [49], and load balancing [50]; etc. In the next section, we elaborate on the research trend to enable the locality-aware capability of structured P2P. All the studied P2P algorithms in this work are assumed to have such functionality. 2.3 Locality-aware Peer-to-Peer Algorithms In most DHT systems, the basic hash functions for the IP address choose peer identifiers randomly and the neighbour relationships are established based solely on these peer identifiers. Locality awareness is not inherent in the design of DHT. As a result, routing stretch of object lookup, i.e. the ratio of the network distance travelled by a lookup query message and the distance between the requesting node and the nearest copy of the target object, can be high. Another challenge of DHT random hashing is load balancing. In a real world corpus, keyword frequency varies. The distribution typically follows Zipf s law [51], meaning that a few keywords occur very often while many others occur rarely. This problem is called a flash crowd or hot spot. These suggest a new research trend to associate the capability of selecting peers in a geographically informed manner. It is called locality-awareness or network proximity in P2P systems Network Proximity in Distributed Hash Tables Structured P2P networks (i.e. DHTs) like [6], [7], [28], [29], [30], [31], [52] offer a novel platform for a variety of scalable and decentralised distributed applications. These systems provide efficient and fault-tolerant routing, object location, and load balancing 19

48 within a self-organising overlay network. One important aspect of these systems is how they exploit network proximity in the underlying Internet. An initial attempt to do so is in the context of CAN system, which was reported by Ratnasamy et al. in [31]. This approach was quite successfully in reducing path latencies. This work proved the important role of the key-space associated with high dimensionality. Reference [53] discusses three basic approaches suggested for exploiting localityaware in DHTs to improve routing performance, including geographic layout, proximity routing and proximity neighbour selection Geographic Layout In geographic layout approach, the identifiers are assigned in a manner that ensures that peers that are close in the network topology are close in the identifier space. In one implementation, peers measure the round-trip time (RTT) between themselves and a set of landmark servers to map the d-dimensional space onto the physical network, such that peers that are neighbours in the d-dimensional space (and therefore in each other s routing tables) are close in the physical network. This technique can achieve good performance but it has the disadvantage that it is not fully self-organising; it requires a set of wellknown landmark servers. In addition, it may cause significant imbalances in the distribution of peers that lead to hotspots. When considering the use of this method in Chord, Tapestry and Pastry, additional problems arise. Whilst geographic layout provides network locality in the routing, it sacrifices the diversity of neighbouring peers in the identifier space, which has consequences for failure resilience and availability of replicated key-value pairs. Both Chord and Pastry have the property that the integrity of their routing fabric is disrupted when an entire leaf set or successor set fails. Likewise, both protocols replicate key-value pairs on neighbouring peers in the namespace for fault tolerance. With a proximity-based identifier assignment, neighbouring peers, due to their proximity, are more likely to suffer correlated failures or to conspire. 20

49 Proximity Routing Proximity routing was first proposed in CAN [31]. The routing tables are built without taking network proximity into account but the routing algorithm chooses a nearby peer at each hop from among the ones in the routing table. Each peer measures the RTT to each neighbour (i.e. routing table entry) and forwards messages to the neighbour with the maximum ratio of progress in the d-dimensional space to RTT. As neighbours are spread randomly over the network topology, the distance to the nearest neighbour is likely to be significantly larger than the distance to the nearest peer in the overlay. Additionally, this approach trades off the number of hops in the path against the network distance traversed at each hop; it may increase the number of hops. Because of these limitations, the technique is less effective than geographical layout. Proximity routing has also been used in a version of Chord [28]. Here, a small number of peers are maintained in each finger table entry rather than one, and a message is forwarded to the topologically closest peer among those entries whose identifier is closer to but counter clockwise from the message s key. Since all entries are chosen from a specific region of the id space, the expected topological distance to the nearest among the entries is likely to be much larger than the distance of the nearest peer in the overlay. Furthermore, it appears that all these entries need to be maintained for this technique to be effective because not all entries can be used for all keys. This increases the overhead of peer joins and the size of routing tables. In conclusion, proximity routing affords some improvement in routing performance, but this improvement is limited by the fact that a small number of peers sampled from specific portions of the identifier space are not likely to be among the peers that are closest in the network topology Proximity Neighbour Selection In proximity neighbour selection approach, routing table construction takes network proximity into account. Routing table entries are chosen to refer to peers that are nearby in 21

50 the network topology, among all live peers with appropriate identifiers. The distance travelled by messages can be minimised without an increase in the number of routing hops. Tapestry and Pastry s locality properties derive from mechanisms to build routing tables that take network proximity into account. They attempt to minimise the distance, according to the proximity metric, to each of the peers that appear in a peer s routing table, subject to the constraints imposed on identifier prefixes. Pastry ensures the following invariant for each peer s routing table: Proximity invariant: Each entry in a peer X s routing table refers to a peer that is near X, according to the proximity metric, among all live Pastry peers with the appropriate identifier prefix. As a result of the proximity invariant, a message is normally forwarded in each routing step to a nearby peer, according to the proximity metric, among all peers whose identifier shares a longer prefix with the key. Moreover, the expected distance travelled in each consecutive routing step increases exponentially, because the density of peers decreases exponentially with the length of the prefix match. From this attribute, two distinct properties of Pastry with respect to network locality are derived: Total distance travelled: The expected distance of the last routing step tends to dominate the total distance travelled by a message. As a result, the average total distance travelled by a message exceeds the distance between source and destination peer only by a small constant value. Local route convergence: The paths of two Pastry messages sent from nearby peers with identical keys tend to converge at a peer near the source peers, in the proximity space, because in each consecutive routing step, the messages travel exponentially larger distances towards an exponentially shrinking set of peers. Thus, the probability of a route convergence increases in each step, even in the case where earlier (smaller) routing steps have moved the messages farther apart. This result has significance for caching applications layered on Pastry. 22

51 The routing algorithms in Pastry and Tapestry allow very effective proximity neighbour selection because there is freedom to choose nearby routing table entries from among a large set of peers. This leads to very good route locality properties. Moreover, the join protocol allows Pastry to identify appropriate nearby peers by performing only a small number of network probes equus: a Locality-aware P2P System In this sub-section, we provide in more detail about a recent research in locality-aware P2P. T. Locher et al. [54] combine several advantages in their proposal of a locality-aware P2P system called equus. They assume that all peers are uniformly distributed in a two dimensional Euclidean space for a simple analysis of system properties. The locality awareness criterion in equus captures the latency of a lookup operation. equus scheme guarantees that the distance travelled by a packet in the network is not much larger than the shortest distance between the source and the destination peer. equus employs a hierarchical hypercube topology. Groups of peers that are close to each other according to the chosen proximity metric form the vertices of a partial hypercube. Within such a group (called clique as it forms a complete graph) all peers share the same identifier, which is a bit string of a predefined length d. The length d of the identifiers is referred to as the dimension of the network. Since these nodes share the same identifier, they are also responsible for the same fraction of the key space. This has two interesting properties. First, these nodes ensure a certain degree of redundancy that is required in case data is lost due to the sudden departure or failure of a particular node. Second, consistency among these nodes can be achieved quickly due to the short distance between those nodes. In addition to the connections to all other clique neighbours, each peer has links to peers in other cliques. These additional links ensure the connectivity of the entire system. Figure 2.4 illustrates an example network of equus. 23

52 Figure 2.4. An example network consisting of 5 cliques. Nodes that are close-by belong to the same clique, share the same ID, and are responsible for the same set of data items [54]. In order to give good locality properties, a newly arriving node must join the closest clique in the system by contacting an arbitrary bootstrap node. The contacted node returns the address of one node of each clique in its routing table. The new node then contacts all those nodes and determines the closest one, e.g. by sending several ping messages and waiting for the replies. This procedure is similar to the mechanism described in [55]. Subsequently, a join message is sent to the closest node which again returns addresses of cliques in its routing table. This step is repeated in O log N rounds until the closest clique has been found. The contacted node in the close clique informs all the other nodes within the clique about the arrival of a new node and gives the new node all the information it needs to become a fully integrated clique member. The routing table can be copied from any other clique member, since they all share links to the same cliques. Since the degree of each node should not exceed O log N to maintain efficient searches, the number of nodes in a clique has to be limited. Once the number of nodes reaches a certain threshold, the clique is confronted with a higher probability of data loss. This observation ensures that, once the number of nodes reaches a certain lower bound, 24

53 the remaining nodes in the clique have to join another clique, resulting a merging of the two cliques. Likewise, if the number of nodes reaches a specific upper bound, this clique has to be split into two cliques. Hence, apart from the standard operations such as JOIN and LOOKUP, two additional queries, namely MERGE and SPLIT, are essential in equus. By using such hierarchical structure, equus also reduces overhead since nodes joining and leaving a clique only trigger communication among the nodes in the same clique. Changes in the routing tables are only necessary if entire cliques appear or disappear, due to the arrival or departure of a large number of nodes. This allows equus to maintain network stability as the life time of cliques is much longer than the lifetime of individual nodes Summary P2P networks benefit from such topological information to improve the neighbour relationship. Peers can gradually select close-by neighbours in the virtual space to achieve much better response times for the lookup operations and reduce routing stretch. It also helps to give them inherently strong resilience to churn. This work is concerned with distributed structured locality-aware P2P systems that are practically applicable to provide VoIP service. Specifically, all ordinary peers in the studied systems should at least have some local awareness about locations and paths to their sets of neighbours, as proposed in the EDR system [8]. This knowledge can be acquired using light-weight measurements such as ping, traceroute, or by querying their local superpeers. Superpeers, on the other hand, are assumed to have full knowledge about the overlay network topology, which should be updated and disseminated periodically [56]. In Chapter 5 of this work, we follow the network model used in equus [54] and GNP [57] to represent P2P topology in 2-dimensional Euclidian space. This simple modelling allows us to study the P2P path tendency, and to sufficiently approximate the diversity degrees of relay paths. 25

54 2.4 Peer-to-Peer VoIP Systems The evolution of P2P technologies realises the capability for end-to-end QoS communication services such as voice and video. P2P systems inherently have high scalability because the capacity scales with user population; robustness and fault tolerance since there is no centralised server and the network self-organises itself. VoIP service can be provided as an application of the P2P network where the VoIP clients form a selforganising P2P overlay network to locate and communicate with others. Compared to other P2P applications, the P2P systems providing this delay sensitive service have the following fundamental differences [35]. Firstly, traditional P2P applications such as file sharing systems provide multiple copies of a popular file. Therefore, the reliability of peers is not a problem. On the other hand, in the case of VoIP, we want to talk to the right person, and not similarly named persons. Furthermore, user contact location may change frequently, as opposed to rather stable directory information and locations for a retrievable file. Secondly, when initiating a call, the caller actively waits for the other side to ring. Therefore, P2P VoIP systems are quite sensitive to lookup latency. On the other hand, other applications like file sharing and directory lookup-based systems can tolerate high lookup latency, and the actual file download time tends to be larger than the lookup latency. Thirdly, voice traffic consumes moderate network bandwidth, e.g. a Skype call typically takes only 3-16 kbps [58]. A voice relay peer, therefore, can handle more connections than a file sharing node does. As a result, load balancing and data storage are generally not issues for VoIP peers. Finally, file sharing and directory services may suffer from flash crowd effects. On the other hand, call access patterns are more uniformly distributed. These differences are summarised on Table 2.1. In the following sub-sections, we study the main performance factors of the P2P VoIP systems impacting on our relay path selection algorithms in this dissertation. We then present some P2P VoIP systems. 26

55 Table 2.1. Summary of different features in P2P applications [35]. Properties/Types File sharing Directory VoIP (for user lookup) Data storage Yes No No Caching Yes Yes No Delay sensitive No No Yes Reliability Having multiple independent copies of data helps Only the intended user must be found VoIP Performance Parameters Generally, the quality of a VoIP call is influenced by three factors: delay, loss and delay variation (or jitter). The International Telecommunication Union (ITU) has defined Mean Opinion Score (MOS) whish is a subjective quality metric to evaluate human feeling speech quality [59]. MOS is given on a scale of 1 (unacceptable) to 5 (excellent) as shown on Table 2.2 below. Table 2.2. The Mean Opinion Score for measuring call quality MOS Rating Perceived Quality 4-5 Excellent Toll quality 3-4 Good Cell phone quality <3 Fair Unacceptable <2 Bad Unintelligible The ITU has also established a method for estimating VoIP call quality from the measured network performances called E-Model [60]. It is computed using the nonlinear function as shown in (2.1), where R is referred to as the R-factor. MOS R R ( R 60 )( 100 R ) (2.1)

56 R-factor is defined as a combination of different aspects of voice quality impairments: R R0 I s I e I d A (2.2) where R0 groups the effects of various noises; Is includes the effect of other impairments that occur simultaneously with the voice signal; Ie covers the impairment caused by different types of losses; Id represents the impairment caused by delays; and, A compensates for the above impairments under various user conditions. In practice, receivers buffers in many VoIP systems are used to smooth out the jitter incurred in transmission environments. Using ITU default values, (2.2) can be reduced to R 94.2 I e I d. (2.3) Therefore, both delay and loss are two factors that need to be considered when designing a VoIP network. Recall from begin of this section that the fundamental differences between VoIP relay paths and paths for other P2P applications are that VoIP paths are sensitive to latency and loss, as well as moderate consumption of network bandwidth. In this work, hence, we do not consider network load as a significant factor. Instead, we consider the degree of path diversity represented by the number of overlap hops between default path and alternate relay path. Given the rich inter-connectivity of the large-scale P2P networks, increasing the degree of divergence is considered to improve end-to-end performance because distant relay paths are unlikely to experience link degradation or failure at the same time. Reference [8] has validated the performance improvement when their proposed algorithm, EDR, utilises such disjointness. Furthermore, by maximising relay path diversity, our overall multi-objective optimisation problem (cf. Chapter 5) can also help improving load balancing because load is distributed among disjoint links and dispersed to a more decentralised set of relay peers. Reference [8] also observed that paths with high delay avoidance percentage are also likely to have high loss avoidance percentage, and vice versa. One of the main reasons might be the fact that paths are experiencing long delay if they are part of a congested part 28

57 of the network, which also causes more losses. We accept this observation in our study to further reduce the complexity. Specifically, we use delay as the second performance metric when comparing different relay path selection schemes Skype Skype System Overview Skype is a P2P VoIP client developed from KaZaA [43] that allows its users to place voice calls and send text messages to other users of Skype clients. In essence, it is very similar to the MSN and Yahoo IM applications, as it has capabilities for voice calls, instant messaging, audio conferencing, and buddy lists. However, the underlying protocols and techniques it employs are quite different. Skype is not an open protocol. All control messages and voice traffic are encrypted. Nonetheless, its successful deployment has attracted much research work. One of the initial studies is the Skype system analysis published by SA. Baset and H. Schulzrinne in [58]. We provide an overview of the Skype system based on this work with a focus on the search functionality of Skype. There are two types of nodes in this overlay network, ordinary hosts and supernodes. An ordinary host is a Skype application that can be used to place voice calls and send text messages. A supernode is an ordinary host s end-point on the Skype network. Any node with a public IP address having sufficient CPU, memory, and network bandwidth is a candidate to become a supernode. An ordinary host must connect to a supernode and must register itself with the Skype login server for a successful login. Although not a Skype node itself, the Skype login server is an important entity in the Skype network. User names and passwords are stored at the login server. User authentication at login is also done at this server. This server also ensures that Skype login names are unique across the Skype name space. Figure 2.5 illustrates the relationship between ordinary hosts, super nodes and login server. 29

58 Apart from the login server, there is no central server in the Skype network. Online and offline user information is stored and propagated in a decentralised fashion and so are the user search queries. Skype login server Message exchange with the login server during login Ordinary host Supernode Neighbour relationships in the Skype network Figure 2.5. Skype network. There are three main entities: supernodes, ordinary hosts, and the login server [58]. Network Address Translation (NAT) and firewall traversal are important Skype functions. Each Skype node uses a variant of the STUN [61] protocol to determine the type of NAT and firewall it is behind, in decentralised manner. Each Skype client builds and refreshes a table of reachable nodes. In Skype, this table is called host cache and it contains IP address and port number of supernodes. This host cache is stored in the operating system s registry for each Skype node. 30

59 Skype claims to have implemented a third generation of P2P, i.e. employing the Global Index (GI) technology [62]. The global indexing strategy is an advanced DHTbased keyword searching scheme. Recall that keyword searching is not directly supported by the DHTs. Hence, the global indexing scheme has to maintain in each node the inverted lists of some keywords, and to assign each undivided keyword to a unique node in the system by hashing to a unique key of DHT layer. An inverted list for a keyword contains all the identifiers of objects in which the keyword appears. To answer a query consisting of multiple keywords, the query is sent to peers responsible for those keywords. Their inverted lists are transmitted over the network and intersected to get a list of objects that contain the keywords [14]. To reduce bandwidth consumption and lookup latency, caching is normally implemented meaning that Skype clients cache the user information sent to them from some queries to avoid receiving them again for future queries Skype Functions Skype functions can be classified into start-up, login, user search, call establishment and tear down. Start-up When Skype client was run for the first time after installation, it sends a Hypertext Transfer Protocol (HTTP) 1.1 GET request to the Skype server (i.e. skype.com). During subsequent start-ups, a Skype client only sends a HTTP 1.1 GET request to the Skype server (skype.com) to determine if a new version is available. Login Login is perhaps the most critical function to the Skype operation. It is during this process a Skype client authenticates its user name and password with the login server, advertises its presence to other peers and its buddies, determines the type of NAT and firewall it is behind, and discovers online Skype nodes with public IP addresses. These newly discovered nodes are used to maintain connection with the Skype network should the supernode to which Skype client is connected become unavailable. 31

60 In login process, Skype client must establish a Transmission Control Protocol (TCP) connection with a supernode in order to connect to the Skype network. If it cannot connect to a super node, it will report a login failure. After a Skype client is connected to a supernode, the Skype client must authenticate the user name and password with the Skype login server. The login server is the only central component in the Skype network. It stores Skype user names and passwords and ensures that Skype user names are unique across the Skype name space. Skype client must authenticate itself with login server for a successful login. After logging in for the first time after installation, host cache is initialised with seven IP address and port pairs. Upon first login, host cache is usually initialised with these seven IP address and port pairs. Thus, they are called bootstrap supernodes. In the case where host cache is initialised with more than seven IP addresses and port pairs, it always contains those seven bootstrap supernodes. It is with one of these IP address and port entries a Skype client establishes a TCP connection when a user uses that Skype client to log onto the Skype network for the first time after installation. For the first time login, A Skype client maintains a TCP connection with at least one bootstrap node to acquire the address of the login server. Skype client then establishes a TCP connection with the login server, exchanges authentication information with it through a challenge-response mechanism. Skype is a P2P client and P2P networks are very dynamic. Skype client, therefore, must keep track of online nodes in the Skype network so that it can connect to one of them if its supernode becomes unavailable. Thus, at the end of the login process, Skype client tries to contact with about 20 distinct nodes to advertise its arrival on the network and to build an alternate node table of online nodes. The subsequent login process is quite similar to the first-time login process. The Skype client builds a host cache after a user has logged in for the first time after installation. The host cache gets periodically updated with the IP address and port number of new peers. During subsequent logins, Skype client uses the login algorithm to determine at least one available peer out of the nodes present in the host cache. It then establishes a TCP connection with that node. 32

61 User Search Skype uses its GI technology to search for a user. Skype claims that search is distributed and is guaranteed to find a user if it exists and has logged in during the last 72 hours. Extensive testing in [58] confirms that the GI guarantees Skype clients to find, within 3-4 seconds, a user who has logged in the Skype network in the last 72 hours. However, the underlying search technique that Skype uses for user search is still not clear. Reference [58] suggests that it uses a combination of hashing and periodic controlled flooding to gain information about online Skype users. Call Establishment and Teardown For users that are not present in the buddy list, call placement is equal to user search plus call signalling. If both users are on public IP addresses, online and are in the buddy list of each other, then upon pressing the call button, the caller host cache establishes a TCP connection with the callee host cache. Signalling information is exchanged over TCP. In the case where the caller is behind port-restricted NAT and callee is on public IP address, signalling and media traffic do not flow directly between caller and callee. Instead, the caller sends signalling information over TCP to an online Skype node which forwards it to callee over TCP. This online node also routes voice packets from caller to callee over UDP and vice versa. If both users are behind port restricted NAT and UDP-restricted firewall, both caller and callee host cache exchange signalling information over TCP with another online Skype node. There are many advantages of having a node route the voice packets from caller to callee and vice versa. First, it provides a mechanism for users behind NAT and firewall to talk to each other. Second, if users behind NAT or firewall want to participate in a conference, and some users on public IP address also want to join the conference, this node serves as a mixer and broadcasts the conferencing traffic to the participants. The 33

62 negative side is that there will be a lot of traffic flowing across this node. Also, users generally do not want that arbitrary traffic should flow across their machines. During call tear-down, signalling information is exchanged over TCP between caller and callee if they are both on public IP addresses, or between caller, callee and their respective host caches Skype Related Research S. A. Baset and H. Schulzrinne [58] also analyse several other key functions of Skype, including NAT/firewall traversal and supernode relay. Following this work, Guha et al. in [63] provide experimental data about Skype system that is useful for future design of P2P VoIP systems, including the population of online Skype clients, the number of supernodes, and their traffic characteristics. Y. Yu et al. in [64] study the Skype system in order to identify P2P traffic and carry out measures on the Skype overlay network with a special ad-hoc tool. Similarly, K. Suh et al. in [65] try to detect and characterise the Skype relay traffic through metrics. In [50], G. Caizzone et al. analyse the scalability aspect of Skype network. It is pointed out that capacity of links is not critical for Skype system scalability. However, it is an issue with the ratio of number of Skype users to the number of supernodes. The supernodes drive the maximum size that the Skype network can reach. The measurement and experiment based studies [56], [66] have demonstrated that the Skype system uses a sub-optimal relay path selection mechanism with large number of unnecessary probes, resulting in heavy network traffic. Reference [56] also suggests through two scenarios that overlay routing paths can be faster (or shorter) than the direct IP routing paths P2P SIP Telephony Skype protocol is proprietary and closed. Another approach for P2P VoIP implementation is based on Session Initiation Protocol (SIP) [67], [68]. SIP is a control protocol originally used in Internet Engineering Task Force (IETF) Internet telephony 34

63 client-server architecture. Its job is to set up, modify, and tear down sessions between session users. The main functions of SIP are to act as a signalling protocol, and to define the type of session for which it is signalling. It can also support sessions via multicast or single unicast, a mesh of unicast sessions, or a combination of these choices. Four major functions that SIP supports are: Determination a user location; Determination of media for the session; Determination of willingness of a user to participate; Call establishment, call transfer, and termination. One of the most important architectural features of SIP is that it relies on client-server model. The majority of the system cost of this model is in maintenance and configuration, typically by a dedicated system administrator in the domain. It also means that quickly setting up the system in a small environment (e.g., for emergency communications or at a conference) is not easy. P2P Internet telephony using SIP [69], [70], [71], [72], [73] has been proposed to avoid the maintenance and configuration cost of the client-server architecture in SIP, and to prevent catastrophic failures of server based systems. The IETF is conducting a working group (i.e. P2PSIP) that develops standards-track specifications for P2P SIP. This effort is based on the use of REsource LOcation And Discovery (RELOAD) Base Protocol [74], a P2P signalling protocol. The P2P signalling protocol provides the network nodes that form an overlay network with abstract storage, messaging, and security services. In this sub-section, we provide a brief introduction to P2P SIP Telephony architectures of the study [35], focusing on the different P2P SIP architectural implementations. There are two approaches for combining SIP and P2P: replace the SIP location service by a P2P protocol (SIP-using-P2P) [69], [72], and additionally, implement the P2P protocol itself using SIP messaging (P2P-over-SIP). In the first case, P2P is used only for lookups and updates of SIP user s IP addresses. A scalable and global P2P location service 35

64 automatically makes the SIP lookups scalable. In the second case, the P2P maintenance protocol can further exhibit two modes: (1) tunnel the P2P protocol messages in SIP, e.g., as a message body or headers, or (2) reuse the semantics of some of the SIP messages and headers to convey proximity and location information. These are fundamentally similar between the two approaches because there is a clear separation between the DHT layer and the SIP layer as shown in Figure 2.6. The difference is that in P2P-over-SIP, the P2P maintenance protocol is also implemented using SIP. Followings are the comparisons of the two architectures. In the SIP-using-P2P architecture, the system can use the optimisations and enhancements done in the external DHT. For example, the message overhead can be reduced for the DHT maintenance. However, the algorithmic overhead of number of messages remains the same and depends on the particular DHT (e.g., Chord) in use. Some SIP specific timers (e.g., retransmission timeout) may not be acceptable for some DHT-based applications, especially if the timers translate to long DHT lookup and update latency. Figure 2.6. Difference between SIP-using-P2P and P2P-over-SIP architectures [35] In the SIP-using-P2P architecture, the node needs to implement the particular DHT connector. If multiple DHTs can be used then such implementations need to potentially implement all such DHT connectors. 36

65 Today, there are multiple P2P protocols that do not interoperate and are not meant to interoperate (e.g., Kademlia, Chord, OpenDHT [75]). Moreover, there is no single protocol or mechanism to talk to any DHT. Thus, the P2P-over-SIP architecture gives us an opportunity to build such an interface using SIP. Using SIP to build the DHT allows us to reuse the existing naming, routing, and security issues from SIP. Moreover, the NAT and firewall traversal mechanisms in SIP can also be used to allow a node behind a NAT to become a super-node. Secondly, SIP features such as redirect and proxy modes are readily reusable in a DHT s iterative and recursive modes. Moreover, we can transparently reuse the existing SIP-based components such as voic and conferencing servers without having them to understand the DHT protocol to update the DHT indicating that they provide the service Summary VoIP has been an active area of research and development in the past decades, with a number of companies providing Personal computer (PC)-to-PC and PC-to-phone calls. Their objective is mainly to provide low-cost call service to Public Service Telephone Network (PSTN) from the public Internet. VoIP system architecture can be either clientserver or P2P. P2P VoIP has the benefit of avoiding high maintenance and configuration cost of the client-server architecture, and preventing the single point of failures problem in server based systems. In this section, we discuss on the important performance parameters of P2P VoIP systems. We then present two architectures of well-known P2P VoIP systems, including Skype and P2P SIP networks. The remarkable impact of Skype system, and consequently, the intense research effort in the field of P2P VoIP have motivated us in this work. 2.5 Relay Path Selection In this section, we introduce several relay path selection schemes that influence our research work, including the Resilient Overlay Network (RON) [76], Detour [77], Path 37

66 Diversity with Forward Error Correction System (PDF) [78], AS-aware Peer relay Protocol (ASAP) [56], and Early Divergence Rule (EDR) [8], [9] Resilient Overlay Network RON [76] is an architecture that allows distributed Internet applications to detect and recover from path outages and periods of degraded performance within several seconds, improving over today s BGP routing protocol that take at least several minutes to recover. A RON is an application-layer overlay on top of the existing Internet routing substrate. The RON nodes monitor the functioning and quality of the Internet paths among themselves, and use this information to decide whether to route packets directly over the Internet or by way of relaying via other RON nodes, optimising application-specific routing metrics. RON's routing mechanism was able to detect, recover, and route around all of them, in less than twenty seconds on average, showing that its methods for fault detection and recovery work well at discovering alternate paths in the Internet. Furthermore, RON was able to improve the loss rate, latency, or throughput perceived by data transfers. RON nodes exchange information about the quality of the paths among themselves via a routing protocol and build forwarding tables based on a variety of path metrics, including latency, packet loss rate, and available throughput. Each RON node obtains the path metrics using a combination of active probing experiments and passive observations of on-going data transfers. Each RON is explicitly designed to be limited in size - between two and fifty nodes - to facilitate aggressive path maintenance via probing without excessive bandwidth overhead. This allows RON to recover from problems in the underlying Internet in several seconds rather than several minutes. The second goal of RON is to integrate routing and path selection with distributed applications more tightly than is traditionally done. This integration includes the ability to consult application- specific metrics in selecting paths, and the ability to incorporate application-specific notions of what network conditions constitute a "fault." 38

67 The third goal of RON is to provide a framework for the implementation of expressive routing policies, which govern the choice of paths in the network. For example, RON facilitates classifying packets into categories that could implement notions of acceptable use, or enforce forwarding rate controls. Path Evaluation and Path Selection in RON RON routers need an algorithm to determine if a path is still alive, and a set of algorithms with which to evaluate potential paths. The responsibility of these metric evaluators is to provide a number quantifying how "good" a path is according to that metric. These numbers are relative, and are only compared to other numbers from the same evaluator. The two important aspects of path evaluation are the mechanism by which the data for two links are combined into a single path, and the formula used to evaluate the path. Every RON router implements outage detection, which it uses to determine if the virtual link between it and another node is still working by using an active probing mechanism. On detecting the loss of a probe, the normal low-frequency probing is replaced by a sequence of consecutive probes, sent in relatively quick succession spaced by PROBE_TIMEOUT seconds. If OUTAGE_THRESH probes in a row elicit no response, then the path is considered "dead". If even one of them gets a response, then the subsequent higher-frequency probes are cancelled. Paths experiencing outages are rated on their packet loss rate history; a path having an outage will always lose to a path not experiencing an outage. The OUTAGE_THRESH and the frequency of probing (PROBE_INTERVAL) permit a trade-off between outage detection time and the bandwidth consumed by the (low-frequency) probing process. RON does not attempt to find optimal throughput paths, but strives to avoid paths of low throughput when good alternatives are available. From the standpoint of improving the reliability of path selection in the face of performance failures, avoiding bad paths is more important than optimising to eliminate small throughput differences between paths. 39

68 Oscillating rapidly between multiple paths is harmful to applications sensitive to packet reordering or delay jitter. While each of the evaluation metrics applies some smoothing, this is not enough to avoid "flapping" between two nearly equal routes: RON routers therefore employ hysteresis. Based on an analysis of 5000 snapshots from a RON node's link-state table, it chose to apply a simple 5% hysteresis bonus to the "last good" route for the three metrics. This simple method appears to provide a reasonable trade-off between responsiveness to changes in the underlying paths and unnecessary route flapping Detour Detour [77] is an overlay framework proposed for path diversity implementation. This mentioned as virtual Internet, in which routers tunnel packets over the public Internet in place of using dedicated links. This design allows easy deployment of experimental infrastructure and, unlike dedicated network testbeds, is subject to real Internet traffic loads. Detour is composed of a set of geographically distributed router nodes interconnected using tunnels. A tunnel can be thought of as a virtual point-to-point link. Each packet entering a tunnel is encapsulated into a new IP packet and forwarded through the Internet until it reaches the tunnel's exit point. This same mechanism has previously been used to form the multicast backbone (MBONE) and the experimental IPv6 backbone (6BONE). Tunnels are useful because they allow new routing functionality to be prototyped while using the existing network infrastructure. A host wishing to use the Detour network will direct its outbound traffic to the nearest Detour router. Its packets will be forwarded along tunnels within the Detour network and will exit at a point close to the destination. In order that responses return in the same fashion, the system must perform network address translation, so the source address of the packet reflects the exit router and not the actual source. This complication is a necessary consequence of using tunnels to superimpose a new routing framework. 40

69 The Detour architecture is illustrated in Figure 2.7. It is important to notice that Detour routers are edge devices and do not appear in the core of the network. By controlling routing and congestion control at the edge of the network, Detour system expects to archive sufficient control while avoid potential problems of supporting perflow processing at the high traffic bandwidths found in the core of the network. Detour routers can exchange information about the measured latency, drop rate and bandwidth available along their tunnels. It also enables the use of dynamic multi-path routing to automatically load balance the Detour system and avoid congestion before it occurs by randomly assigning flows to good paths and by dynamically varying how traffic is spread across such paths. Furthermore, Detour architecture allows routers to be able to classify traffic, and select a routing policy best suited to the needs of each traffic class. Figure 2.7. Architecture of the Detour virtual Internet [77] In summary, Detour system enables multipath routing at the router level whereas other P2P relay path selection schemes considered in this section accomplish this task at application level. 41

70 2.5.3 PDF Path diversification system with forward error correction (PDF) [78] has been proposed for a single sender, single receiver over packet switched networks such as the Internet. PDF system is similar to RON [76] in that it consists of a set of participating nodes that receive and forward packets to other nodes. However, PDF forwards packets simultaneously over multiple redundant paths rather than selecting an optimal path to send all the packets on. The central question to be solved in PDF in one sender one receiver scenario with path diversity is whether there exists sufficiently disjoint paths between a pair of senders and receivers on the Internet to result in uncorrelated loss patterns between paths. If the paths are not entirely disjoint, then the probability of congestion on the shared links between the paths must be small in order to minimise the overall end-to-end loss. PDF project has further characterised the disjointness between the redundant and default paths for various Internet topologies, and show significant reduction in packet loss using PDF system over single path scheme. To set up a communication channel between the sender and receiver, the sender first executes traceroute from itself to all the participating nodes and the receiver. The information returned from the traceroute includes the link latencies, and the names of the routers along the default path from the sender to the receiver and all paths between the sender and the participating nodes. The sender also sends a setup packet to all participating nodes instructing them to execute traceroute from themselves to the receiver. Next, all the participating nodes send the path information between themselves and the receiver obtained from traceroute to the sender. The sender now has the names of the routers and their associated link latencies for the default path, the paths between itself to all participating nodes, and the paths between participating nodes to the receiver. This information is used to compute the optimal redundant path. After the redundant path via a chosen relay node is selected, the sender sends a setup packet to the selected relay node, instructing it to forward packets to the receiver on behalf of the sender from then onward. The setup packet contains a flow ID, IP address, and the 42

71 port number of the receiver. Upon receiving the setup packet, the relay node stores an entry containing a flow ID, an IP address, and a port number of the receiver in a table. This table is used to forward packets on behalf of different senders to their receivers. Each packet sent from a different sender to the relay node contains a different flow ID in its header. The relay node forwards a packet to the right IP address and port number of the receiver based on its flow ID. Note that all the setup messages and executions of traceroute are done only at the start of the session. In delay sensitive applications, PDF system chooses to only use one relay node to forward packets between a pair of sender and receiver. The reason is that the delay of a redundant path via multiple participating nodes in series is likely to be larger than that of the redundant path via only one node. However, PDF can be easily extended to the case where there are multiple redundant paths via multiple relay nodes. Intuitively, the path selection scheme first finds a set of redundant paths that are as disjoint as possible from the default path. Within this set of redundant paths, it then selects the one that results in minimum latency. An alternative might seem to be to select the redundant path based on traffic characteristics of each link along the path between two nodes. However, PDF does not have knowledge of loss rates and bandwidths for individual links, thus, it chooses the redundant path to be maximally disjoint with the default Internet path so as their losses are uncorrelated. This results in PDF system to be effective in minimising the packet loss rate. PDF system does not aggressively send out probing packets to monitor the current loss rates and bandwidths between participating nodes. Instead, it simply uses the route information between participating nodes, and only invokes traceroute at the start of the session. Hence, the solution is scalable, even though its performance is potentially lower than that of RON with aggressive probing for network conditions. One of the drawbacks of PDF system is that its performance depends on the information provided by traceroute. This information can be incomplete or inaccurate. For example, traceroute can only differentiate between routers and not switches. 43

72 Another drawback is that some ASs do not report accurately or deliberately hide information about their networks; only certain routers are visible to outside and therefore, complete information is not available AS-Aware Peer-relay Protocol S. Ren et al. have proposed an AS-Aware Peer-relay Protocol (ASAP) in [56]. ASAP shows that by having knowledge of AS topology, P2P VoIP systems can yield better voice call performance than the AS-unaware versions of Skype system. It proposes a complete solution for building such large VoIP overlay system. The design of ASAP is based on the following Internet properties. (1) In general, peer nodes with the same IP prefix are relatively close to each other [79]. The collection of all peers with the same IP prefix form a cluster. The direct IP routing latency between two peers in two different clusters can be estimated by the direct IP routing latency between any pair of nodes in their corresponding clusters. (2) With publicly available BGP tables and updates (e.g., [80], [81]), an up-to-date annotated AS graph can be built. (3) The number of AS hops and the latency of a direct IP routing path are correlated, and paths with longer AS hops are likely to have longer latency [82]. (4) An Internet AS-level direct IP routing path usually has the valley-free property [83]. ASAP defines the three types of nodes, bootstrap, cluster surrogate, and normal end hosts. The ASAP system structure is illustrated in Figure 2.8. The operations of the node types are as follows. Bootstraps are normally the powerful, dedicated, and always-on servers used to process VoIP user login and nodes join requests. They provide following functions and services to make the entire system informative and intelligent: 1. Build an annotated up-to-date AS graph. 2. Build an IP prefix to cluster surrogate IP mapping table and an IP prefix to AS number (ASN) mapping table. Upon the join request of a new node, translate the node IP to its ASN and its cluster surrogate IP, return the ASN and the cluster surrogate IP to the new node. Note that an AS can have multiple IP prefixes. 44

73 3. Disseminate the AS graph to surrogates, so that every surrogate has an up-to-date AS graph. 4. Select new surrogates for clusters upon surrogate failures. Bootstrap1 Bootstrap s data structure Internet AS graph Bootstrap2 IP prefix to cluster Surrogate IP table Cluster surrogate s data structure Cluster s close cluster set IP prefix to ASN table Bootstraps Surrogate SA Internet AS graph Surrogates Cluster s top node table Cluster C Surrogate SB Cluster A End host h3 Cluster B End host h1 End hosts End host h2 Figure 2.8. ASAP system structure [56]. Surrogates are powerful and stable with high bandwidth network connections within a cluster. These nodes volunteer themselves to provide the following services: 1. Maintain the list of IP addresses of all end hosts in their clusters. 2. Periodically contact bootstrap nodes to retrieve the up-to-date annotated AS graph. 3. Periodically run an algorithm to construct close cluster sets for their clusters (The close cluster set of a cluster are those clusters whose end hosts have short direct IP routing latencies to any end host in this cluster). 4. Process close cluster set requests from other end hosts in their clusters. 45