Guaranteeing Performance through Fairness in Peer-to-Peer File-Sharing and Streaming Systems Alex Sherman Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2010
c 2010 Alex Sherman All Rights Reserved
ABSTRACT Guaranteeing Performance through Fairness in Peer-to-Peer File-Sharing and Streaming Systems Alex Sherman Over the past decade, Peer-to-Peer (P2P) file-sharing and streaming systems have evolved as a cheap and effective technology in distributing content to users. Guaranteeing a level of performance in P2P systems is, therefore, of utmost importance. However, P2P file-sharing and streaming applications suffer from a fundamental problem of unfairness, where many users have a tendency to free-ride by contributing little or no upload bandwidth while consuming much download bandwidth. By taking away an unfair share of resources, free-riders deteriorate the quality of service experienced by other users, by causing slower download times in P2P file-sharing networks and higher stream updates miss rates in P2P streaming networks. Previous attempts at addressing fair bandwidth allocation in P2P, such as BitTorrent-like systems, suffer from slow peer discovery, inaccurate predictions of neighboring peers bandwidth allocations, under-utilization of bandwidth, and complex parameter tuning. We present FairTorrent, a new deficit-based distributed algorithm that accurately rewards peers in accordance with their contribution in a file-sharing P2P system. In a nutshell, a FairTorrent peer uploads the next data block to a peer to whom it owes the most data. FairTorrent is resilient to exploitation by free-riders and strategic peers, is simple to implement, requires no bandwidth over-allocation, no prediction of peers
rates, no centralized control, and no parameter tuning. We implemented FairTorrent in a BitTorrent client without modifications to the BitTorrent protocol, and evaluated its performance against other widely-used BitTorrent clients using various scenarios including live BitTorrent swarms. Our results show that FairTorrent provides up to two orders of magnitude better fairness, up to five times better download performance for contributing peers, and 60-100% better performance on average in live BitTorrent swarms. We show analytically that for a number of upload capacity distributions, in an n-node FairTorrent network, no peer is ever owed more than O(log n) data blocks with high probability. Achieving fair bandwidth allocation in a P2P streaming scenario is even more difficult, as it comes with an additional constraint: each stream update must be received before its playback deadline. P2P live streaming systems require global resource over-provisioning to deliver adequate streaming performance. When there is not enough bandwidth to accommodate all users for a particular stream, such as due to free-riders or low-contributing peers, all users, including high-contributing peers, observe poor performance. We present FairStream, a new P2P streaming system that delivers a good quality stream to peers that upload data at a rate above the stream rate, even in the presence of free-riders or malicious users. FairStream achieves this with three mechanisms. First, it provides a new peer reply policy framework that enables file sharing incentive mechanisms to be adapted for streaming. Second, it uses this framework to incorporate a deficit-based peer reply policy that enables each peer to reply first to the neighbor to whom it owes the most data as measured by a deficit counter. Third, it introduces a collusion-resistant mechanism to ensure effective data distribution of a stream despite a large fraction of free-riders who do not forward received data. We prove that FairStream is resilient to free-riders and rewards peers with streaming performance correlated with their contributions. We
have also implemented FairStream as a BitTorrent client and evaluated its performance against other popular streaming systems. Our results on PlanetLab show that FairStream, similar to other systems, provides good quality streaming performance when resources are over-provisioned, but it also provides orders of magnitude better streaming performance for peers uploading above the stream rate when resources are constrained, in the presence of free-riders and low-contributing peers.
Contents Contents i List of Figures vi List of Tables xi 1 Introduction 1 1.1 Peer-to-Peer File-Sharing and FairTorrent............... 2 1.2 Peer-to-Peer Streaming and FairStream................. 5 1.3 Novel Contributions............................ 9 2 Background and Related Work 13 2.1 Overview of BitTorrent.......................... 14 2.2 Fairness in BitTorrent-like Systems................... 15 2.3 Reputation and Credit-based Systems.................. 18 2.4 Fairness in Packet Flow Allocation................... 19 2.5 Structured vs Unstructured Streaming Systems............ 20 2.6 Fairness in P2P Streaming........................ 21 2.7 Streaming Server............................. 24 3 FairTorrent Algorithm for Peer-to-Peer File-sharing 26 i
3.1 Definition of Fairness........................... 26 3.2 Algorithm................................. 30 3.2.1 Leecher Behavior......................... 30 3.2.2 Seed Behavior........................... 37 3.2.3 Exchanging Data......................... 38 3.2.3.1 Unchoking Peers.................... 38 3.2.3.2 Reaching Rate Convergence.............. 38 3.2.3.3 Data Availability.................... 40 3.2.3.4 Bootstrapping...................... 40 3.2.3.5 TCP friendliness.................... 40 3.3 FairTorrent Properties.......................... 43 3.3.1 Measure of Fairness........................ 43 3.3.2 Fast Rate Convergence...................... 44 3.3.3 High Utilization and Incentives................. 45 3.3.4 Resilience to Strategic Peers................... 45 4 Analysis, Modeling and Simulation of the FairTorrent Algorithm 47 4.1 Problem Specifications, Terminology and Assumptions........ 49 4.1.1 Problem Specification and Terminology............. 49 4.1.2 Assumptions............................ 50 4.1.3 Objective............................. 51 4.2 A Simple Distribution.......................... 52 4.3 Uniform Distribution........................... 61 4.3.1 Modeling a Load Queue as a Markov Chain.......... 62 4.3.2 Modeling the Arrival Rate.................... 62 ii
4.3.3 Steady-State Behavior...................... 67 4.3.4 Completing The Model...................... 68 4.3.5 Working with the Model..................... 70 4.3.6 Verifying the Model........................ 71 4.3.7 Bounds on the Maximum Deficit................ 76 4.4 General Distribution and Bounds.................... 77 4.4.1 Bounds on the Maximum Deficit................ 78 4.4.2 Steady-State Behavior...................... 81 4.5 Simulation Results............................ 81 4.5.1 Network Size........................... 82 4.5.2 Peer s Upload Capacity...................... 83 4.5.3 Experiment Duration....................... 85 4.5.4 Network Delay.......................... 87 4.6 Comparison with Fair Scheduling.................... 88 4.6.1 Fair Scheduling Algorithms................... 88 4.6.2 FairTorrent Applied to Scheduling................ 95 5 Evaluation Methodology 99 5.1 Node Selection.............................. 102 5.1.1 Generating a List of High Capacity Machines......... 102 5.1.2 Selecting Machines for an Experiment.............. 104 5.2 Running the Experiments........................ 105 5.2.1 Ensuring Synchronization via Phased Deployment....... 105 5.2.2 Detecting Stale Clients via White List Distribution...... 106 5.2.3 Test Monitoring, Termination and Log Collection....... 107 iii
5.3 BitTorrent-specific Deployment Features................ 108 5.3.1 Using Local Torrents....................... 108 5.3.2 Cleaning Dot Files........................ 108 5.4 Configuration of the Experiments.................... 109 5.5 Analyzing Experimental Results..................... 109 5.6 TestDeploy Components......................... 110 6 FairTorrent Evaluation 112 6.1 Uniform Distribution........................... 113 6.2 Skewed Distribution........................... 128 6.3 Dynamic Live Distribution........................ 129 6.4 Live Swarms................................ 134 7 FairStream: Peer-to-Peer Live Streaming with Resilience to Free- Riders and Low-Contributing Peers 137 7.1 Introduction................................ 137 7.2 System Design............................... 139 7.2.1 Request and Buffering Policy.................. 141 7.2.2 FairStream Reply Policy..................... 142 7.2.3 Preemptive Re-requests...................... 144 7.2.4 Streaming Server......................... 146 7.2.5 Best-Peers Optimization..................... 147 7.2.6 Additional Parameters...................... 151 7.3 FairStream Properties.......................... 151 7.3.1 The Case of No Excess Bandwidth............... 152 iv
7.3.2 Assuming Excess Capacity.................... 157 7.3.3 Resilience to Strategic peers assuming Excess Bandwidth... 158 8 FairStream Evaluation 159 8.1 Global Overprovisioning......................... 162 8.2 Two Upload Classes........................... 163 8.3 Flash Crowd and Dynamic Arrivals................... 165 8.4 A Large Range of Upload Classes.................... 170 8.5 Free-Riding................................ 174 9 Conclusions and Future Work 175 9.1 Our Results................................ 175 9.2 Implications................................ 177 9.2.1 Reliable Services......................... 177 9.2.2 Open Streaming.......................... 178 9.3 Future Work................................ 179 9.3.1 Fairness of P2P Data-Exchange................. 179 9.3.2 Application of FairTorrent.................... 180 Bibliography 182 v
List of Figures 3.1 FairTorrent algorithm for leechers L 1, L 2 and L 3 with upload capacities of 3, 2 and 2................................ 35 3.2 Peers L 1, L 2 and L 3 with upload capacities of 3, 2, and 2. The figure shows bandwidth allocations in blocks/s between these peers over the first four 10-second intervals under four different systems: (a) Fair- Torrent, (b) BitTorrent s equal-split, (c) Block-based TFT and (d) PropShare. For Block-based TFT we assume a threshold limit of 2 data blocks................................. 37 4.1 Markov chain modeling the size of the load queue for peer L i..... 63 4.2 Progress made by peer L i in sending load packets to the peers on its randomized list. The figure captures a possible progression in time from top to bottom............................ 64 4.3 Deficit lists of several peers: L i, L u, L v, L w, L x captured at a point in time. Existing holes are annotated with peers they correspond to. L m is responsible for the largest number of holes.............. 65 4.4 Expected Load on Peers with Upload Capacities between 1 and 10.. 73 vi
4.5 Expected Load on Peers with Upload Capacities between 1 and 50.. 74 4.6 Expected Load on Peers with Upload Capacities between 1 and 100. 75 4.7 SameRate upload capacity distribution: Maximum positive and negative deficits across all peers in a SameRate distribution for networks of size 50, 100, 200 and 400. The error bars capture the minimum, average and the maximum values of the max deficits recorded across 20 experiments.............................. 84 4.8 Uniform upload capacity distribution: Maximum positive and negative deficits across all peers in a uniform distribution for networks of size 50, 100, 200, and 400. The error bars capture the minimum, average and the maximum values of the max deficits recorded across 20 experiments 84 4.9 Exp upload capacity distribution: Maximum positive and negative deficits across all peers in a Exp distribution for networks of size 50, 100, 200, and 400. The error bars capture the minimum, average and the maximum values of the max deficits recorded across 20 experiments 85 4.10 Maximum Deficit observed by nodes of a given upload capacity over 20 experiments in a network of 200 nodes with a uniform distribution of upload capacities............................ 86 4.11 Maximum deficit observed over time in three different experiments under the SameRate, the Uniform and the Exp distributions of the upload capacities.................................. 87 4.12 Maximum deficit over time for a network of 200 nodes under the Uniform upload capacity distribution for 4 different network delay values of 0, 100, 200 and 300 milliseconds.................... 89 vii
4.13 GPS: scheduling packets for flow S1 of weight 10 and S2 through S11 of weight 1 under GPS. Each flow is allocated its proportional bandwidth. 91 4.14 WFQ: scheduling packets for flow S1 of weight 10 and S2 through S11 of weight 1 under WFQ. Flow S1 gets ahead by 10 packets before any packets from other flows are scheduled. p 10 1 and p 11 1 mark the 10th and the 11th packets sent to the flow S1................. 92 4.15 WF2Q: scheduling packets for flow S1 of weight 10 and S2 through S11 of weight 1 under WF2Q. Flow S1 gets ahead by at most 1 packet ahead of the GPS-based allocation.................... 93 6.1 Uniform: FairTorrent fairness....................... 116 6.2 Uniform: BitTorrent fairness........................ 117 6.3 Uniform: Azureus fairness......................... 117 6.4 Uniform: PropShare fairness........................ 118 6.5 Uniform: BitTyrant fairness........................ 118 6.6 Uniform: rate convergence to 90% capacity for 40-50KB/s leechers.... 119 6.7 Uniform: rate convergence to 90% capacity for all leechers........ 119 6.8 Uniform: E max + and Emax......................... 120 6.9 Uniform: High-uploaders download time................. 120 6.10 Uniform: Download time of all the leechers................ 121 6.11 Uniform: FairTorrent download time.................... 121 6.12 Uniform: BitTorrent download time.................... 122 6.13 Uniform: Azureus download time..................... 122 6.14 Uniform: PropShare download time.................... 123 6.15 Uniform: BitTyrant download time.................... 123 viii
6.16 Uniform: FairTorrent utilization...................... 125 6.17 Uniform: BitTorrent utilization...................... 125 6.18 Uniform: Azureus utilization........................ 126 6.19 Uniform: PropShare utilization...................... 126 6.20 Uniform: BitTyrant utilization....................... 127 6.21 Uniform: Standard deviation of the download rate............. 127 6.22 Skewed: Download time.......................... 130 6.23 Skewed: high uploader E max +, Emax.................... 130 6.24 Dynamic: high uploaders average download times............. 133 6.25 Dynamic: high uploaders E max +...................... 133 6.26 Live download rate............................. 135 8.1 Miss Rate for well-provisioned networks................. 166 8.2 Miss rate with two upload classes.................... 166 8.3 Median miss rate vs buffer time..................... 167 8.4 95th percentile miss rate vs buffer time.................. 167 8.5 FlashCrowd: initial high-uploaders................... 168 8.6 FlashCrowd: high-uploaders that arrive................. 168 8.7 Dynamic arrivals: initial high-uploaders................. 169 8.8 Dynamic Arrivals: high-uploaders that arrive.............. 169 8.9 PL: Delivery Rate vs. Upload Capacity................. 170 8.10 GM: Delivery Rate vs. Upload Capacity................ 170 8.11 BT: Delivery Rate vs. Upload Capacity................. 171 8.12 PS: Delivery Rate vs. Upload Capacity................. 171 8.13 FS: Delivery Rate vs. Upload Capacity................. 172 ix
8.14 Miss rate of high-uploaders mixed with 33% low-uploaders and 33% free-riders................................. 172 x
List of Tables 3.1 FairTorrent terminology.......................... 31 4.1 Values computed by the model for the expected load: N, and the expected number of holes: H on a network with 200 nodes with upload capacities chosen from the uniform distribution............ 72 4.2 Maximum positive and negative service errors for different scheduling algorithms, and run-time per packet in terms of the number of flows: n 90 4.3 Packets sent by and received by FairTorrent peer S0 from its peers through time. Each cell (Si, DF 0i (t )) indicates the deficit value, DF 0i (t ), observed by S0 from peer Si right before the start of the time unit t. A cell may contain letters r or s, corresponding to a packet that was received by or sent by S0 to a given peer right before the start of a time unit................................ 97 xi
Acknowledgments This dissertation is dedicated to my parents, Marina and Boris Sherman, who encouraged me throughout the program. I thank my adviser Jason Nieh who encouraged me to work on a problem of interest to me, helped me formulate the problem and supported me throughout the doctoral program. I also thank Cliff Stein for many discussions related to the theoretical analysis of the problem. xii
Chapter 1 Introduction Peer-to-Peer (P2P) file-sharing and streaming systems have evolved as a cheap and effective way to distribute content to users. In the past decade, the usage of P2P file-sharing applications on the Internet has experienced explosive growth. Many individuals and businesses now rely on P2P file-sharing for distributing videos, software, and documents. More recently, P2P video streaming systems have also been growing in popularity. Many P2P streaming systems [52, 69, 32] boast hundreds of thousands of concurrent users and transmit sport games, concerts, and other events. While leveraging individual users upload bandwidth allows P2P systems to deliver content in a cheap and scalable manner, these systems often suffer from poor performance. In particular, free-riders, users who contribute little or no bandwidth, are prevalent in file-sharing systems and take away significant resources from the systems, causing slow downloads for other users. Even a high-contributing peer cannot guarantee its own performance in the presence of free-riders. Similarly, in a P2P streaming system, a high-contributing user that may be contributing bandwidth many
Chapter 1. Introduction 2 times above the rate of the stream, can observe a poor streaming quality with high miss rate, or high percentage of missed stream updates, in the presence of users that contribute little bandwidth. This dissertation addresses the problem of the vulnerability of P2P systems in the presence of free-riding and low-contributing users. We develop algorithms that allow users to receive fair reciprocation from the system. By fair reciprocation we refer to a scenario where a peer receives the download bandwidth from its neighbors that closely matches the peer s own upload bandwidth contribution. Because of the fair reciprocation, we demonstrate that, both in P2P file-sharing and P2P streaming, a high-contributing user can be guaranteed a high quality performance despite the presence of a large number of free-riding and low-contributing users in the system. 1.1 Peer-to-Peer File-Sharing and FairTorrent Although P2P file-sharing is now an integral part of our computing experience, filesharing applications are plagued by a fundamental problem of unfairness in how bandwidth among peers is used and allocated: peers do not receive service commensurate with what they contribute to the system. Unfairness causes many performance problems. Peers are disincentivized from contributing more upload bandwidth. A growing number of free riders, peers who cap their upload bandwidth to zero or a small value, take as much as possible from the system while contributing little resources. On the other hand, high-contributing peers often observe slow download times in the presence of free-riders. Thus, a peer cannot guarantee their own performance by increasing contribution. While there has been much research in fair bandwidth allocation to a set of network flows by a router [16, 47, 20, 7, 58], fair bandwidth allocation in P2P systems can
Chapter 1. Introduction 3 be difficult to achieve for several reasons. First, bandwidth resources are distributed and belong to a number of peers, rather than a single entity such as a router. Unlike a router or a server, no central entity controls and arbitrates access to all resources. Second, the amount of bandwidth resources available is not known in advance and peers cannot be relied upon to specify their own resources honestly. Third, bandwidth resources may vary over time for several reasons, including changing network conditions, peers joining and leaving the system, mobile peers connecting at different access points, and users using available bandwidth for other activities. Finally, some peers, such as free-riders who intentionally cap their bandwidth to zero or a very small value, may try to take advantage of the system, Any fair allocation mechanism must be strong enough to withstand attempts by free-riders and strategic peers to manipulate the system. Many approaches have attempted to address the problem of fair bandwidth exchange in file-sharing systems. There are three main approaches. The most common approach, employed by the popular file-sharing system BitTorrent [13], splits a peer s upload bandwidth equally among a subset of neighboring peers and adjusts this subset based on estimates of their bandwidth rates. This approach is commonly referred to at the tit-for-tat (TFT) heuristic. A second approach is block-based TFT [9, 27, 63], used by BitTyrant [49] peers among one another, which augments TFT with hard limits on the amount of data one peer can owe another. A third approach is embodied by PropShare [31] in which a peer s upload bandwidth is split among its neighbors in proportion to past estimates of its neighbors bandwidth contributions. All of these approaches are rate-based and suffer from a fundamental flaw. They assume that neighboring peers rates measured over some arbitrary period of time provide accurate capacity estimates and are indicators of future bandwidth contributions. In practice, this assumption is problematic as both P2P bandwidth availability and
Chapter 1. Introduction 4 consumption can vary substantially from one time interval to another. This results in significant problems with existing approaches, including unfairness, exploitation by strategic clients [59, 49, 39], bandwidth underutilization [9, 27], ad-hoc parameter tuning requirements, and poor performance. We present FairTorrent, a new deficit-based distributed P2P algorithm that solves the problem of fair bandwidth exchange in the presence of free-riders and strategic peers. FairTorrent accurately rewards peers in accordance with their contribution. In its simplest form, FairTorrent, which runs locally at each peer, maintains a deficit counter for each neighbor which represents the difference between bytes sent and bytes received from that neighbor. When it is ready to upload a data block it sends it to the peer with the lowest deficit, or simply the peer to whom it owes the most data. Unlike other approaches, FairTorrent uses a completely different mechanism that does not rely on any explicit estimates of peer bandwidth, thereby avoiding the bandwidth rate estimation problem that has plagued other approaches. By selecting the destination of the next block to go to the neighbor with the smallest deficit, FairTorrent always acts to minimize the difference between bytes sent and bytes received, thereby minimizing unfairness at each step. The result is a high degree of fairness, and performance for each peer closely correlated with its contribution. FairTorrent provides several key advantages over other approaches: (1) It provides fair bandwidth allocation, operating only at individual peers, in a distributed manner that does not require any centralized control of peers or other P2P resources. (2) It is provably resilient to free riders and other strategic clients. (3) It allows a peer to maximize its upload capacity utilization. (4) It avoids long peer discovery and reaches a fast rate convergence; i.e. it quickly obtains a bandwidth reciprocation rate from its neighbors equal to its own contribution. (5) It does not need to estimate and predict peers allocations, allocate precise upload or download rates for any peers,
Chapter 1. Introduction 5 or rely on advanced knowledge of available bandwidth of other peers. (6) It has no magic parameters, requires no tuning, and is simple to implement. A FairTorrent client requires no changes to the BitTorrent protocol, making it compatible and easy to use with existing BitTorrent clients. We have implemented FairTorrent inside a BitTorrent client and evaluated its fairness and performance on the PlanetLab [51]. We compared against four other widely used open-source BitTorrent implementations, the original BitTorrent Python client by Bram Cohen [13], the popular Azureus Java BitTorrent client [6], the Prop- Share [31] client that claims to provide proportional bandwidth allocation, and the strategic BitTyrant [49] client. Our results show that FairTorrent outperforms the other clients across a wide range of different client bandwidth distributions, static and dynamic scenarios, as well as in live swarms. 1.2 Peer-to-Peer Streaming and FairStream Peer-to-Peer (P2P) video streaming systems have been growing in popularity. Many P2P streaming systems, including PPlive [52], GridMedia [69], and CoolStreaming [32], boast hundreds of thousands of users and transmit sport games, concerts, and other events including the US President s Inauguration [26]. However, while commercially-sponsored events such as the Inauguration, that was streamed from the CNN s website, typically provide good streaming quality due to additionallyprovisioned server bandwidth, unsponsored streams often have spotty performance for all users. The problem is that none of the P2P live streaming systems contain incentives mechanisms, that either prevent free-riders (i.e. users that contributing little or no bandwidth) from deteriorating performance of contributing users or mechanisms that reward users that contribute more bandwidth with a better quality stream. As
Chapter 1. Introduction 6 a result, when there is insufficient bandwidth to accommodate everyone all users observe poor streaming quality regardless of their bandwidth contribution. To make matters worse, all of the aforementioned streaming clients are closed-source and do not provide an easy way for a user to limit their upload bandwidth contribution. Thus, when the bandwidth provisioning for a stream is low, the streaming client can easily take all of the available upload bandwidth from the users who already contribute a lot, and cripple other network-bound processes running on the users machines. Of course, on the other extreme, if the software were to allow the users to configure their bandwidth contribution, without any incentives mechanism, the P2P streaming systems would invite a lot of free-riding and degrade performance even further. Free-riding, which is a common problem in P2P file-sharing systems, would be even more disastrous in P2P streaming. While in most file-sharing systems users would prefer faster downloads they would sometimes be satisfied with eventual download completion. When a user watches a stream in real-time any drop in the download rate results in missed packets and decreased quality of the stream. Thus, for a system without incentives, increase in free-riding means higher likelihood of poor quality for all users. In addition to the drop in the average per-peer capacity, free-riding significantly limits the effectiveness of a streaming server in distributing the stream. Since the server does not know apriori who the free-riding peers are, even if it is provisioned with ample upload capacity it may distribute some updates exclusively to free-riding peers. Since the free-riders do not forward stream updates they receive, some updates may never get distributed to contributing peers in time. This problem, is especially difficult in the streaming scenario. While in file-sharing, the peers can always request missing file blocks at a later time, in streaming, each update has to be distributed to the users prior to its playback deadline.
Chapter 1. Introduction 7 We show, in Chapter 6, that all prior file-sharing incentives mechanisms, including BitTorrent s TFT, PropShare [31], that implements proportional response, and blockbased TFT [9], provide weak incentives mechanisms even in the context of file-sharing, where a steady reciprocation rate needs not be maintained. Because of the more stringent constraints for streaming, the mechanisms such as BitTorrent s tit-for-tat, are even less suitable for streaming than they are for file-sharing. In streaming, a user must be guaranteed a steady reciprocation rate at least equal to the rate of the stream. Even a temporary drop in the download rate may cause missed stream updates and, thus, poor stream quality. The slow and imperfect peer discovery of BitTorrent s tit-for-tat [49] make it a poor candidate for a reliable mechanism to guarantee a steady reciprocation rate. As a result of the added difficulty, the existing proposals in P2P streaming fail to guarantee a fair reciprocation rate. Some proposals, along the lines of Substream- Trading [38] attempt to evaluate peer s reciprocation rates over timed rounds and suffer from similar shortcomings as BitTorrent s TFT. Other proposals, such as BAR Gossip [34] and its successor, FlightPath [33] enforce pseudo-random peer pairings over short rounds. However, an unlucky peer may require many additional pairings to obtain necessary data and the bandwidth usage may spike as much as 100%, requiring a significant room for the overhead. To address these problems, we introduce FairStream, a new P2P streaming system that delivers a good quality stream to peers that upload data at a rate above the stream rate, even in the presence of free-riders or malicious users. FairStream achieves this with three key mechanisms. First, it provides a framework that introduces the notion of a peer reply policy, enabling incentive mechanisms from P2P file sharing to be applied to streaming for the first time. A reply policy determines the order in which a peer replies to pending requests, while a request policy determines the order
Chapter 1. Introduction 8 in which a peer requests data. Previous work has largely ignored reply policies and instead has focused on request policies. Our framework combines reply policy incentive mechanisms with existing streaming optimizations for effective dissemination of stream updates, ensuring that peers have data to exchange with one another so that they can effectively utilize their upload bandwidth. Second, FairStream demonstrates the critical importance of a fair reply policy as an incentive mechanism for streaming. It leverages the FairTorrent deficit-based algorithm used for P2P file-sharing in its peer reply policy. Just like in FairTorrent, in FairStream, a peer keeps track of its deficit with each neighbor, which is the bytes sent to minus bytes received from each neighbor. For its reply policy, a FairStream peer replies to the peer with the smallest deficit, which is the peer to whom it owes the most data. This deficit mechanism fosters very precise data exchange rates even with streaming playback deadline constraints. The new reply policy is resilient to free-riders and strategic peers, and ensures that a peer uploading at a rate above the stream rate will receive an adequate stream download rate from the system. Each peer makes its decisions locally, resulting in a fully-distributed fairness mechanism. Third, FairStream introduces a collusion-resistant mechanism to ensure effective data distribution of a stream by a streaming server despite a large fraction of freeriders who do not forward received data. A peer does not just request from the server data that it is missing, but instead it requests data that is also missing from among its best quality peers, where quality is measured based on how quickly it can download data from these peers. This mechanism enables the server to effectively distribute data chunks missing in the peer s neighborhood, even in the presence of free-riders. To the best of our knowledge, we are the first to tackle this problem that presents a particularly acute threat in P2P streaming systems. We implemented our FairStream client on top of the existing BitTorrent client. We
Chapter 1. Introduction 9 evaluated FairStream on the PlanetLab and compared it against the heuristics used by popular P2P streaming systems, PPlive [24] and GridMedia [69], as well as incentive mechanisms used in common file-sharing P2P systems, including BitTorrent s tit-fortat and ProportionalResponse [31]. We show a number of scenarios where FairStream allows peers contributing at just above the stream rate to view the stream with a minimal miss rate despite the presence of a large number of free-riders and lowcontributing peers. 1.3 Novel Contributions This dissertation considers the challenge of implementing strong real-time fairness in P2P file-sharing and streaming systems, a problem that has long existed in unmanaged, P2P systems. A system that could fairly reward a peer would also guarantee a high quality of performance to a high-contributing peer. In this dissertation we propose a novel fair algorithm, its analysis, evaluation, and application to both, filesharing and streaming scenarios. Novel contributions of this dissertation include the following: 1. The Introduction of a Novel P2P Algorithm, FairTorrent. We introduce a new distributed algorithm, FairTorrent, that guarantees a high degree of real-time fairness of bandwidth exchange among peers in a P2P file-sharing network, and thus, guarantees a fast download rate for high-contributing peers. FairTorrent runs a simple and elegant algorithm where it always sends the next data block to a peer to whom it owes the most data. FairTorrent avoids the pitfalls of rate-based allocation approaches as it does not need to explicitly estimate neighbors rates or explicitly allocate rates to its neighbors. As a result, FairTorrent is able to obtain a very fast and much more precise convergence of
Chapter 1. Introduction 10 its upload rate to the reciprocation rate that it receives from its peers. FairTorrent runs in a fully distributed manner, and requires no central coordinator to regulate bandwidth exchange, and no third party service to maintain or store virtual credit as in credit-based systems. 2. FairTorrent Implementation. We implement a FairTorrent BitTorrent client on top of both, the original python-based BitTorrent client, and the popular java-based Azureus client. The algorithm requires no changes to the BitTorrent protocol. We demonstrate the efficacy and compatibility of FairTorrent by evaluating it both inside a closed network as well as joining live BitTorrent swarms. 3. Analysis of FairTorrent. We analyze the fairness of a n-node FairTorrent network, where the upload capacities of the peers are selected from various distributions. We analyze the fairness in terms of the worst-case number of data blocks that could be owed to or owed by a peer at a given time. We show for a number of upload capacity distributions that no peer owes or is owed more than O(log n) data blocks at any time with high probability. We show that for a general case, even including a highly-skewed distribution of upload capacities there is a bound of 2(n 1) on the blocks owed to or owed by a peer. 4. Evaluation Methodology. We present the deployment and evaluation methodology of FairTorrent on the PlanetLab [51]. In particular, we present a deployment process that uses staged deployment, and white lists and guarantees a high-degree of robustness of our experiments despite the irregularities and variable uptime of the underlying PlanetLab nodes. Unlike prior work, we also show that it is important to normalize live network results by the peer degree, or number of peer connections parameter, in order to differentiate the effect of
Chapter 1. Introduction 11 a given bandwidth exchange algorithm. 5. Comparison and Evaluation of FairTorrent We evaluate and compare the performance and fairness of the FairTorrent client against other commonly-used open-source BitTorrent implementations on both the PlanetLab, and in public BitTorrent swarms. We compare against four BitTorrent implementations: the original BitTorrent Python client by Bram Cohen [13], the popular Azureus Java BitTorrent client [6], the PropShare client [31] that implements the proportional bandwidth allocation [68], and the strategic BitTyrant [49] client that implements a k-tft heuristic between BitTyrant clients. We evaluate these clients in static networks under various distributions of peer upload capacities, dynamic settings with joins and departures of peers, and by joining live BitTorrent networks. We also evaluate a skewed distribution where a high-uploading peer contends with free-riders and low-contributing peers. Finally we evaluate the performance of high-uploading clients of one type inside networks running other types of clients. Our results show that FairTorrent provides up to two orders of magnitude better fairness, up to five times better download times for contributing peers, and 60% to 100% better performance on average in live BitTorrent swarms. 6. The Use of a Reply Policy as a Method to Introduce Incentive Mechanisms in P2P Streaming We introduce the use of the peer reply policy, or the algorithm used by a peer to handle request queues from neighboring peers, as a mechanism to implement alternative incentives mechanism in a P2P streaming system. While state-of-the-art P2P systems largely ignore the peer reply policy, and by default reply to requests in first come first served (FCFS) order, we implement incentives mechanisms by varying the algorithm used to
Chapter 1. Introduction 12 handle requests from peers. Varying the reply policy also allows us to compare various incentives schemes. 7. Design of a Resilient P2P Live Streaming Client We design and implement a P2P live streaming system, FairStream, that provides good performance to high-contributing peers in a streaming scenario despite the presence of many low-contributing and free-riding peers. Our design achieves this property by modifying the reply policy to implement an incentives mechanism based on FairTorrent, where a peer handles pending streaming requests in the order of increasing deficits. 8. Collusion-Resistant Algorithm to Deal with Free-riders We introduce a collusion-resistant mechanism inside FairStream to ensure effective data distribution of a stream by a streaming server despite a large fraction of free-riders who do not forward received data. 9. Comparison and Evaluation of FairStream. We evaluate FairStream on the PlanetLab under a number of distributions of peer upload capacities and scenarios that include many free-riding and low-contributing peers. We compare the evaluation of FairStream with other documented P2P streaming solutions as well as other incentives mechanisms used in P2P file-sharing. We show that for peers who upload at a rate slightly above the rate of the stream, FairStream limits the stream updates miss rate to two orders of magnitude smaller as compared to other streaming systems and incentives mechanisms despite the presence of many free-riding and low-contributing peers.
Chapter 2 Background and Related Work This chapter describes prior work that has been done in the area of fair bandwidth exchange in P2P file-sharing and streaming systems. We begin with the overview of Bit- Torrent, the most popular P2P file-sharing application. Then, we review some works that have studied the effectiveness of BitTorrent s so called Tit-for-Tat (TFT) mechanism, that attempts to address fair bandwidth exchange. We review BitTorrent-based derivatives, including block-based TFT and proportional rate allocation, that attempt to improve the fairness characteristics of BitTorrent. We also review other approaches that have proposed to address fairness in P2P file-sharing such as reputation-based and credit-based systems. We then comment on the similarities between fair bandwidth allocation problems in P2P file-sharing and fair flow allocation in routers, a very well-studied problem in the research community. Finally, we turn to P2P streaming. We review both the widely-used P2P streaming systems and research-based systems. While most widely-used P2P streaming systems focus on overall performance, rather than fairness, some recent research has proposed the use of techniques like substreamtrading and proof-of-misbehavior messages to address fairness in P2P streaming. We also review the bandwidth provisioning requirements placed on the streaming servers
Chapter 2. Background and Related Work 14 by the P2P streaming systems. 2.1 Overview of BitTorrent In BitTorrent [13], peers participating in the download of the same target file form a swarm. The target file is conceptually broken up into pieces (typically 256 KB). Peers tell one another which pieces of the target file they already have and request missing pieces from one another. Requests are typically made for 16 KB sub-pieces. Peers that already have the entire file are called seeds. Peers that are still downloading pieces of the file are called leechers. Upon joining a network a peer contacts a tracker, that maintains a list of peers participating the the swarm. The tracker returns a set of up to 50 randomly selected peer addresses (or <IP, port> pairs). After joining the swarm a peer contacts the tracker periodically, every 10-20 minutes. Each peer tries to maintain a set of max peers TCP connections with other peers in the swarm. (50 in Azureus, 80 in BitTorrent). This set represents the neighbors of the peer. Each peer exchanges certain protocol information with her neighbors. For instance, when a peer downloads a new file piece it sends a HAVE message to all of its neighbors with the index number of the piece it downloaded. BitTorrent employs a tit-for-tat (TFT) heuristic to incentivize peers to upload and attempts to provide fair exchange of bandwidth between peers. In each 10-20 second round BitTorrent selects up to N peers from among its neighbors to upload blocks to. N k of the peers have provided the best download rate during the most recent round, and k peers are randomly selected to help discover other peers with better upload rates. (N is typically between 4 and 10, and k is typically between 1 and 2). Thus, the active set of peers, or the peers to which a client currently
Chapter 2. Background and Related Work 15 uploads, is updated in each round based on measurements of their download rates. BitTorrent refers to the selection and deselection of a peer for uploading as unchoking and choking, respectively. During each the round a peer serves requests for sub-pieces to the N unchoked neighbors in a round-robin fashion. The pieces of a file are indexed, and leechers request pieces based on a rarest-first heuristic. Under this heuristic a peer always requests a piece that is least-commonly available (or has the fewest copies) among its neighbors. Because the peer always tries to get pieces that are least-commonly available, this policy is very effective in making sure that a peer always has some pieces that are of interest to its neighbors. Since the peers almost always have something to push to their neighbors, their upload capacity is utilized very effectively. 2.2 Fairness in BitTorrent-like Systems Because of its popularity, much work has been done in studying BitTorrent s behavior. BitTorrent peers tend to exchange data with other peers with similar upload rates over a large file download [30]. Under some bandwidth distributions, the system has been shown to eventually converge to a Nash equilibrium [53]. However, there is no evidence that this behavior extends to shorter file downloads, dynamic environments, skewed distributions of users, or modified but compatible BitTorrent clients. In fact, several modified BitTorrent clients [39, 59, 49] have been developed which exploit different strategies to achieve better performance at the expense of users running unmodified BitTorrent. For example, BitTyrant [49] exploits the fact that BitTorrent will reciprocate at a higher rate even when receiving much smaller bandwidth from BitTyrant in return. The previous studies demonstrate that BitTorrent s TFT heuristic does not result
Chapter 2. Background and Related Work 16 in fair bandwidth exchange. Because TFT only identifies and exchanges data with a small number of peers at a time, a BitTorrent client may waste much time and bandwidth while discovering peers with similar upload rates in a large network [49]. Further waste occurs because relationships with discovered peers may be unstable, as the other peers are also always searching for better peers. Even after discovering peers with good upload rates, BitTorrent continues to blindly donate a portion of bandwidth by randomly uploading to new peers in hopes of reciprocation. Block-based TFT [9] (BB-TFT) attempts to improve on BitTorrent by changing TFT to allow a client to upload to a larger set of neighbors at the same time, but stopping the upload to a peer once its deficit, the difference between what it uploaded to the peer and what it downloaded from that peer, exceeds a certain threshold. While FairTorrent [55] and BB-TFT appear similar in that both use peerwise deficits, their usage of deficits is fundamentally different. FairTorrent uses deficits to decide deterministically where to send the next data block. By sending the next block to a peer to whom FairTorrent owes the most data, it always acts to minimize unfairness and thereby converges quickly to fair bandwidth exchange with its peers. BB-TFT uses deficits only to impose a threshold, but still uses TFT for scheduling. Merely stopping one peer from uploading to another when a threshold is reached does not by itself help peers reach fair rate reciprocation. The hard threshold causes underutilization of the peers upload capacities [9], as high-uploading peers may stop uploading to other peers once they reach the limit. To compensate, a tracker could attempt to match peers with similar bandwidth by trusting peers to report their capacities honestly [9]. This is problematic in practice as peers could game the system by lying about their bandwidth. While a limited simulation study shows that BB- TFT can improve the fairness of BitTorrent [9], another study shows that BB-TFT has poor performance and bandwidth utilization compared to BitTorrent [27]. Our
Chapter 2. Background and Related Work 17 experiments with BitTyrant [49] peers, which use BB-TFT when exchanging data with each other, also show that BitTyrant clients do not receive a fair exchange rate and suffer from underutilization. SWIFT [63] augments BB-TFT with a heuristic to have peers donate a small fraction of their bandwidth, but it is not clear how to tune their highly-parametrized algorithm in a realistic deployment scenario. PropShare [31] attempts to achieve fair bandwidth allocation by implementing the Proportional Response algorithm [68] on top of the Azureus client. Using the algorithm, a peer splits its upload rate to its neighbors for a given round in proportion to the contributions received from them in the previous round. Under simplistic assumptions, the theoretical work in [68] shows that the algorithm converges to a market equilibrium, and a peer receives fair reciprocation after O(log n) rounds, in a n-node network. The algorithm assumes that each peer can accurately estimate all of its neighbors upload rate allocations towards it in each round; peers require these estimates to readjust their own allocations correctly in the next round. This assumption is problematic in practice. Very long rounds would be required to allow each peer to exchange enough data with all of its neighbors to obtain estimates. However, estimates over long rounds would be of limited usefulness for determining allocations under dynamic conditions that occur in practice. PropShare attempts to address this problem by estimating allocations over four exponentially-weighted 10-second rounds. Because peers allocations change and PropShare uploads to and estimates rates of only a small subset of neighbors in each round, PropShare fails to create an accurate view of the current rate allocations of a larger neighborhood. Moreover, the peerwise convergence process is often interrupted and reset because PropShare clients divert 20% of their bandwidth in each round to explore new peers, much like BitTorrent. Our experimental results, in Chapter 6, show that PropShare demonstrates poor rate convergence and does not reward contributing peers fairly.
Chapter 2. Background and Related Work 18 In contrast, FairTorrent s new deficit-based approach does not need to explicitly set upload rates and avoids the pitfalls associated with having to estimate peers changing rate allocations, dealing with insufficient and ad-hoc round durations, and needing to explore a peer s entire neighborhood. It may be possible to tune PropShare to use very small blocks in order to help it explore a larger neighborhood faster, as required by [68]. However, using small blocks would significantly increase the overhead associated with BitTorrent and IP protocols. Some work has explored tradeoffs between performance and fairness in BitTorrent. Based on the assumption that leechers leave the system upon completion of download, one model proposes to optimize average performance by lowering the download rate for high uploaders to keep them in the system longer [18]. However, this assumption is not realistic as many leechers remain in the system as seeds in many BitTorrent systems [11]. Furthermore, other work suggests that fairness does not need to come at the expense of performance [64]. 2.3 Reputation and Credit-based Systems Other approaches have explored different aspects of improving fairness in non-bittorrent P2P systems. Reputation-based systems [28, 35, 8, 14] attempt to separate good contributors and free riders by associating reputations with peers to make it less likely that peers with good reputations will exchange data with free riders. Such systems suffer from problems with bootstrapping and collusion [54], where malicious peers can hype one another s reputation. Even if a perfect reputation metric can be established, such systems do not provide a mechanism for translating reputation into a highly-fair bandwidth-sharing service. BAR [1], BAR Gossip [34], and FlightPath [33] propose to stem out selfish users by relying on signed proof-of-misbehavior messages. BAR