Nearcast: A Locality-Aware P2P Live Streaming Approach for Distance Education

Nearcast: A Locality-Aware P2P Live Streaming Approach for Distance Education XUPING TU, HAI JIN, and XIAOFEI LIAO Huazhong University of Science and Technology and JIANNONG CAO Hong Kong Polytechnic University 7 Peer-to-peer (P2P) live video streaming has been widely used in distance education applications to deliver the captured video courses to a large number of online students. By allowing peers serving each other in the network, P2P technology overcomes many limitations in the traditional clientserver paradigm to achieve user and bandwidth scalabilities. However, existing systems do not perform well when the number of online students increases, and the system performance degrades seriously. One of the reasons is that the construction of the peer overlay in existing P2P systems has not considered the underlying physical network topology and can cause serious topology mismatch between the P2P overlay network and the physical network. The topology mismatch problem brings great link stress (unnecessary traffic) in the Internet infrastructure and greatly degrades the system performance. In this article, we address this problem and propose a locality-aware P2P overlay construction method, called Nearcast, which builds an efficient overlay multicast tree by letting each peer node choose physically closer nodes as its logical children. We have conducted extensive simulations to evaluate the performance of Nearcast in comparison with the existing RTT and NICE protocols. Also, Nearcast has been deployed on a wide-area network testbed to delivery video coursed to about 7200 users distributed across 100 collages in 32 cities in China. The experimental results show that Nearcast leads to lower link stress and shorter end-to-end latencies compared with the RTT and NICE protocols. Categories and Subject Descriptors: K.3.1 [Computers and Education]: Computer Uses in Education Distance learning; C.2.2 [Computer-Communication Networks]: Network Protocols Applications (SMTP, FTP, etc.); H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems Video (e.g., tape, disk, DVI) This work was supported in part by the China National Natural Science Foundation (NSFC) under grant Nos. 60703050, 60642010, and 60433040, and by the NSFC/RGC Joint Research Foundation under grant 60731160630. Authors addresses: X. Tu, H. Jin, and X. Liao, Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, 430074, China; email: {xptu,hjin,xfliao}@hust.edu.cn; J. Cao, Department of Computing, Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong; email: csjcao@comp.polyu.edu.hk. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or permissions@acm.org. C 2008 ACM 1533-5399/2008/02-ART7 $5.00 DOI 10.1145/1323651.1323653 http://doi.acm.org/ 10.1145/1323651.1323653

7:2 X. Tu et al. General Terms: Algorithms, Design, Experimentation, Performance Additional Key Words and Phrases: Peer-to-peer, distance education, live streaming, video ACM Reference Format: Tu, X., Jin, H., Liao, X., and Cao, J. 2008. Nearcast: A locality-aware P2P live streaming approach for distance education. ACM Trans. Intern. Tech. 8, 2, Article 7 (February 2008), 23 pages. DOI = 10.1145/1323651.1323653 http://doi.acm.org/10.1145/1323651.1323653 1. INTRODUCTION Video streaming has been widely used for delivering contents for distance education applications [Zhang et al. 2005a; Liu et al. 2006]. However, in a system with the traditional client/server architecture, to support a large number of remote students taking online courses at the same time is a great challenge. The video servers cannot provide enough bandwidth and even CPU cycles to handle millions of requests with acceptable delay. The peer-to-peer (P2P) computing technology can be used to overcome these difficulties by sharing the resources, for example, bandwidth, and computing resources, available at all the server and client nodes. In P2P media streaming systems, multimedia contents are delivered to a large pool of distributed users with low delay and high robustness. Therefore, P2P has become one of the leading technologies in distance education to deliver the captured video courses. In our previous work, we have developed a P2P media streaming system called Apple for delivering live video courses to remote online students [Jin et al. 2004]. However, Apple does not perform well when the number of the online students increases. The major reason is that the method for constructing the peer overlay in Apple does not consider the topology mismatch problem. The mechanism for peers to randomly choose logical nodes to connect does not take into account any knowledge about the underlying physical network topology. As a result, this can cause serious topology mismatch between the P2P overlay network and the physical network [Liu et al. 2004]. The topology mismatch problem brings great link stress in the Internet infrastructure and greatly degrades the system performance. Link stress [Chu et al. 2001] is the number of redundant copies of a packet carried by a network communication link. In this article, we describe an improvement to Apple which provides a locality-aware P2P overlay construction method called Nearcast. To address the topology mismatch issue, Nearcast builds an efficient overlay multicast tree by letting each peer node choose physically closer nodes as its logical children. The captured video course content is delivered from the root of the overlay tree. Each node on the tree can receive the media data from its upper-level nodes to play back while relaying it to the lower-level nodes. Our analysis and experimental results show that Nearcast can greatly reduce the link stress and end-to-end latency compared with the NICE protocol and approach using RTT to measure the distance between two nodes. The rest of this article is organized as follows. In Section 2, we describe related work. In Section 3, we describe the preliminaries and assumptions

Nearcast: Locality-Aware P2P Live Streaming Approach 7:3 used throughout this article. We describe the Nearcast approach in detail in Section 4. Simulation and deployment evaluation are reported and discussed in Sections 5 and 6, respectively. Finally, Section 7 ends the article with our concluding remarks and a description of our future work. 2. RELATED WORK Several P2P systems for e-learning have been proposed [Brito and Moura 2001]. HYDRA [Zualkernan 2005] is a light-weight, SCORM-based P2P e-learning architecture. It is built on JXTA and uses a fast native XML database for sharing, searching, and downloading e-learning objects. However, HYDRA focuses on the framework of e-learning applications without covering how to distribute efficiently the media content among the users. The APPLE system [Jin et al. 2004] was the first e-learning application that applied P2P live streaming technology. As described earlier, Apple did not take into consideration the topology mismatch problem in its overlay construction. In the following, we survey existing work on P2P live streaming that deal with topology mismatch in this the system design. Network-level IP multicast [Deering and Cheriton 1990] was proposed over a decade ago for delivering content to a group of distributed users. However, it still remains very difficult to deploy widely. The P2P approach, also called application level multicast, has attracted much interest in both research communities and industry. In recent years, we have seen more and more work on providing P2P live video services [Andersen et al. 2001; Castro et al. 2003; Chawathe 2000; Hefeeda et al. 2003; Kostic et al. 2003; Liao et al. 2006; Zhang et al. 2005b; Padmanabhan et al. 2002; Ratnasamy et al. 2001; Rowstron and Druschel 2001]. It has been recognized that a P2P media streaming system should be scalable so it can support many hosts, possibly in excess of hundreds or even millions, under dynamic user arrival and departure. To improve the scalability and to reduce the average link stress in the overlay network, several approaches have been proposed. End System Multicast (ESM) [Chu et al. 2001] uses a mesh-based overlay structure to broadcast video streams to a small group of users. ESM solves the topology mismatch issue by optimizing the overlay after a full mesh connecting all the peers is built. This produces high overhead and, as a result, the system can support only a very limited number of users. In DagStream [Liang and Nahrstedt 2006], the system first constructs an overlay and then optimizes it at run-time by continuously cutting those inefficient connections and adding promising connections. However, it is not clear how the effect of the dynamics introduced by cutting the connections can be avoided. Unlike DagStream, Nearcast proposed in this article organizes the peers according to their proximity at joining time rather than at run-time. The approach used in NICE [Banerjee et al. 2002] and Zigzag [Tran et al. 2003] first organizes the peers into a hierarchy of clusters such that nearby nodes are grouped into the same cluster. Then it builds the overlay data delivery path from the cluster leaders to the other members recursively. The

7:4 X. Tu et al. disadvantage of this approaches is that it is difficult to accurately select powerful nodes with high outgoing bandwidth capacity to be the cluster leaders while guaranteeing the network proximity. Other P2P systems, such as ALMI [Pendarakis et al. 2001], CoopNet [Padmanabhan et al. 2002], TAG [Kwon and Fahmy 2002], and Overcast [Jannotti et al. 2000], use the tree-based method to construct overlay. In particular, ALMI use a centralized method while Coopnet and Overcast use a distributed method in a parent selection algorithm to find close nodes. Centralized parent selection cannot easily adapt to large-scale networks [Shi et al. 2001; Sitters 2002]. CoopNet divides the data into multiple strips and constructs multiple, diverse distribution trees to distribute them, but to maintain multiple trees is a difficult work and introduces much maintenance overhead. In Overcast, peers measure the bandwidth to a parent candidate and determine whether it is the proper parent; with this method it is also difficult to find a lightweight and accurate bandwidth measuring tool. TAG uses route overlapping as a heuristic in constructing a low-delay overlay tree in a distributed manner. It is really a clever method except that it needs the support of the underlying routers to find out the routing path between any two nodes using TraceRoute-likely tools. Our proposed Nearcast also uses the tree-based method, but it differs from the above systems in that it takes into account the locality issue by assigning a well-designed, prebuilt coordinate to each peer through which it measures the distance and selects a close parent. Nearcast also considers the real-life network connectivity constraint issue, which is not addressed by any of the overlays already mentioned. 3. PRELIMINARIES AND ASSUMPTIONS Before describing the detailed design of Nearcast, in this section we introduce the preliminaries and assumptions used throughout this article. We consider that the physical network topology is a multilayer hierarchical (H) multicast tree, which is rooted at a single source and comprises of end hosts and directed data delivery edges. Each end host has the following attributes: Network position. Like in the Global Network Position system (GNP) [Ng and Zhang 2002], in the hierarchical physical network model, we assign a unique H-dimension coordinate to identify the network position of a vertex. Then a node X is assigned a coordinate given by a (H 1) tuple {C 1 (X ),..., C H 1 (X )}. The ith layer network distance between X and Y, denoted by D i (X, Y), is defined by the following formula: D i (X, Y ) = C i (X ) C i (Y ). (1) Connectivity constraint. The end hosts can be categorized into three types: (1) open host. 1 which has no connectivity constraint; (2) half-freerider, such as the hosts behind NATs, which accept only the connections started from 1 We use the terms node, end host, student, and user interchangeably.

Nearcast: Locality-Aware P2P Live Streaming Approach 7:5 the hosts behind the same NAT; (3) full-freerider, such as the hosts behind personal firewalls, which reject all the connection requests from other end hosts. Limited upload bandwidth. Each open host X (or half-freerider) is willing to share a certain amount of upload bandwidth, denoted by B(X). The design of the Nearcast approach is based on the following assumptions: (1) Each end host is aware of its own network position coordinate and connectivity constraint. Methods for generating the network position coordinates and detecting the connectivity constraints of the end hosts will be discussed in Section 6.1. (2) The last hop(s) to an end host exhibits (exhibit) the lowest delay. The design of Nearcast has not considered a long-latency link, for example, a satellite link, as the last hop. (3) For the latencies on the links between the nodes in the hierarchical physical network, the triangular inequality 2 [Guyton and Schwartz 1995] holds. 4. OVERLAY CONSTRUCTION AND MAINTENANCE IN NEARCAST The main idea of the proposed Nearcast approach is as follows. The end hosts of the the application layer overlay multicast tree are first self-organized, based on their network position coordinates, into an H-layer hierarchy of subtrees. Then the captured video courses are broadcast along the multicast tree to each of the user nodes on the tree. The overlay organized in this way has a better match in topology to the underlying physical network so that the link stress and the total (or average) end-to-end delay can be effectively reduced. In this section, we first describe how the overlay tree is organized and maintained. Then we present an analysis of the performance of Nearcast. Finally, we discuss the deployment considerations of Nearcast. 4.1 Tree Organization Corresponding to the physical network model, the overlay constructed in Nearcast is an H-layer hierarchy of subtrees. The overlay is built based on the following rules. Rule 1. A subtree at layer i (0 < i H 1), denoted by s i, comprises the end hosts with the same values for their j th (i < j H 1) layer coordinates but different values for their ith layer coordinates. A subtree at layer 0 comprises the end hosts with the same coordinate values for all layers. Rule 2. A super node of a subtree s i (0 < i H 1) is the parent node of the root of s i, which should be the nearest to the supernode among all the hosts of s i. 2 Geometrically, the triangle inequality states that the sum of latency of any two sides of a triangle is greater than the latency of the remaining side, and the difference of any two sides of a triangle is less than the latency of the remaining side, where the triangle is formed by three fully connected network nodes.

7:6 X. Tu et al. Fig. 1. Part (a) shows an example of three-layer hierarchy of subtrees on the overlay multicast tree, where the captured video courses are broadcast from the source S and transmitted along the overlay tree to the users. The physical structure of the nodes is shown in Figure 2(a). Part (b) shows the three-dimensional network coordinate values of all the hosts in (a). Fig. 2. (a). An example of freerider support. (a) Physical network; (b) IP addresses of some hosts in Rule 3. A subtree at layer 0 is organized randomly except that half-freeriders should always select the host with the nearest public IP address to become their parent. Figure 1(a) illustrates an example of the three-layer hierarchy of subtrees, while Figure 1(b) shows the network position coordinates of the hosts. In this example, hosts H 2, H 3, and H 4 have the same second layer network coordinate values as the source host S, that is, C 2 (H 2 ) = C 2 (H 4 ) = C 2 (H 3 ) = C 2 (S) = 0. However, their first layer coordinate values are different from each other. Hence, they are organized into a subtree B at layer 1 and the source S is the supernode of B. Figure 2 illustrates another freerider example of intrasubtree organization. Referring to Figure 2(a), H 12 and H 13 are grouped into the same subtree severed by H 6 since they are from the same NAT. The public IP addresses of H 6, H 12, and H 13 are listed in Figure 2(b).

Nearcast: Locality-Aware P2P Live Streaming Approach 7:7 Fig. 3. Examples of (a) simply admit operation, (b) insert operation, and (c) redirect operation. 4.2 Tree Management To effectively construct and maintain the Nearcast tree, an end host should maintain a small amount of state information, including IP addresses and port numbers, layer numbers, connectivity constraints, and its network position coordinates, its parent and grandparent, the source host, and its children hosts. The layer number of an end host X, denoted by h(x), represents the order of the layer at which X is located in the Nearcast tree. The protocol for tree maintenance includes the operations for the host to join and depart from the overlay. 4.2.1 Host Join. Initially, the Nearcast tree contains a single subtree at layer H 1 with only the data source host. The layer number of the source node is initialized as H 1. H is determined in the design period and, normally, it equals to the height of the hierarchy. In our implementation of Nearcast, the hierarchy comprises country, ISP, city, postcode, IP 1,IP 2, and IP 3, where IP i means the ith bytes of the IP. So the height of the hierarchy is set to 7. A new open-host X knows the source node via accessing a well-known rendezvous point (RP), and then begins its join process by sending a Join message to the source node. The Join message contains its network coordinates and connectivity constraints. Once an existing host Y receives the Join message from X, it executes the admission algorithm (to be discussed below) to determine whether it is the nearest host to X. If so, it should admit X as a child. Otherwise, it redirects X to the next nearest node among its children. More specifically, one of three basic operations may be performed by Y: (a) admit X as a child; (b) insert X between Y and an existing child of Y; and (c) redirect X to a nearer host Z among the children of Y. In the last case, X should resend the Join message to Z. The process repeats until X finds its nearest parent. Figure 3 illustrates some examples of these three basic operations. The pseudocode of the admission algorithm executed in a node Y is shown in Figure 4. Y first determines whether it is located in a subtree at the lowest layer 0. If Y is located at the layer 0 (i.e., h(y) = 0), and has adequate bandwidth to serve the joining host X(B (Y) > B(X)), Y will admit X, else it simply redirects X to an open-host Z randomly selected from its children. Otherwise, one of the following two cases will occur. Case 1. C h(y ) (X ) = C h(y ) (Y ). In this case, based on the overlay organization Rule 1, X should belong to a subtree located at a lower layer. Thus, Y tries to

7:8 X. Tu et al. Fig. 4. The pseudocode of the admission algorithm. find a child Z at its lower layer and redirect X to it. This policy lets Y reserve certain capacity to accept other peers satisfying C h(y ) (X) C h(y ) (Y). If Z does not exist in the children set of Y, X will be simply assigned to layer h(y ) 1 and admitted as a new child of Y. IfZ is not an open-host, X is inserted between Y and Z. Otherwise, Y should compute and compare the h(y ) 1 layer network distances D h(y ) 1 (Y, Z) and D h(y ) 1 (Y, X) to determine whether the new host X should be inserted between Y and Z or redirected to Z. Case 2. C h(y ) (X) C h(y ) (Y). In this case, X and Y should belong to the same subtree at layer h(y). Thus, Y computes and compares the ith layer

Nearcast: Locality-Aware P2P Live Streaming Approach 7:9 Fig. 5. An example of the host joining process; the network coordinate values of the existed hosts are illustrated in Figure 1(b). network distances from X to all the children of Y to find a child Z satisfying D h (Y ) (Z, X) < D h (Y ) (Y, X). If Z exists, Y simply redirects X to Z. Otherwise X should be admitted as a child of Y. In this scenario, Y tries to find a child Z that is the nearest to Y among all the children of Y. IfZ exists, X should be inserted between Y and Z. Otherwise, Y simply admits X as a child. Figure 5 shows an example of the join process. The network coordinate of the new host X is (1, 2, 3) and the network coordinates of the existing hosts are illustrated in Figure 1(b). In this case, X first sends a Join message to the source node S. Since D 2 (X, S) > D 2 (X, H 1 ), S redirects X to the host H 1. After H 1 receives the Join message from X, it finds that C 2 (H 1 ) = C 2 (X ) = 1 and D 1 (X, H 1 ) > D 1 (H 5, H 1 ). So it redirects X to H 5. The process repeats until X finds its nearest parent H 11. 4.2.2 Host Departure. We propose methods to handle a host departure that occurs gracefully or accidentally. A graceful departure occurs when a host X intends to leave the overlay, while an accident departure occurs when X fails. The failure of X can be detected by its children since the video data stream would be interrupted. For the graceful leaving procedure, X sends out Leave messages to both its parent and children. If a child receives the Leave message, it immediately sends a Rejoin message to its original grandparent. If a child detects that its parent has failed, it immediately sends out a ParentLeave message to its original grandparent. Recovery from the departure of a host at layer 0 is trivial. The parent of the departured host randomly admits a grandchild as its new child and redirects other grandchildren to the newly admitted one. We consider the recovery process for the departure of a host X located at a high layer. Suppose that the parent of X is P and 2 h (P) H 1. Once P receives Leave and Rejoin messages (or ParentLeave messages), it first determines whether X has a child Y located at layer h(x) 1. If Y is not in the overlay tree, then P simply admits the nearest grandchild to become its new child and redirects others to the new child. Otherwise, the leaf-promotion procedure described below is performed. In the leaf-promotion procedure, the parent of the departured host intends to promote a nearest leaf node (or offspring) to become its child. Figure 6 shows

7:10 X. Tu et al. Fig. 6. An example of the leaf-promote procedure. an example of this procedure. In Figure 6, the parent P first sends a Probe message downward to the child R of X whose layer number is h(x ) 1. Once R receives this message, it forwards the message to its nearest child. The procedure repeats until a leaf F or a host at layer 0 is reached. If a host at layer 0 receives the Probe message, it randomly chooses a child to continually propagate that message until the final leaf host F is reached. Then F immediately sends a Response message to P and becomes P s child. Finally, P redirects the children of the departed X to the promoted child F. Note that if the Probe message reaches a host whose children are all half-freeriders or full-freeriders, then that host should be the promoted leaf. At the same time, it should redirect its children to its current parent or to P if the current parent is the departed X. 4.2.3 Freerider Support. The join and leave operations of half-freeriders and full-freeriders are similar to those of open hosts except for the following two points. When a half-freerider joins a subtree at layer 0, it should be redirected to the host with the nearest public IP address. This allows the admission of more half-freeriders because the hosts behind the same NAT can be grouped into the same branch. In the admission algorithm, if Y has already had a half-freerider Z, and receives a Join message from a new half-freerider X (or full-freerider), Y first determines whether X and Z are behind the same NAT and whether it still has available upload bandwidth to serve additional freeriders. If X and Z are behind the same NAT, X should be redirected to Z. IfX and Z are behind different NATs (or X is a full-freerider) and Y still has enough upload bandwidth to serve additional freeriders, X is simply admitted to one of those additional branches. Otherwise, X should be rejected. 4.2.4 Asymmetric Bandwidth and Dynamics Supports. In a practical system, many peers cannot receive enough data for playback because the peer nodes may join and leave the overlay frequently and have very limited upload bandwidth B(X). We propose a patch scheme to solve the problem. A peer X to be accepted as a child will receive a neighbor list from its parent. If the peer finds that it cannot receive enough data from its parent, it sends a Branch-Join (to differentiate from the previous Join ) request to join another peer in the neighbor list so as to get an additional data provider. X will be also added to

Nearcast: Locality-Aware P2P Live Streaming Approach 7:11 the neighbor lists of the selected peer. How to select the peer as the additional provider depends on the available bandwidth and current playtime of the candidates. The peer with its current playtime older than the branch-joiner that has enough or higher available upload bandwidth B(X) will be selected. Suppose the peer just receives 75% of the total media data. It then negotiates with its original parent to ask it to only deliver segments with sequence ID NOT equal to 4k, and let the newly joined branch-parent send segments with sequence ID equal to 4k. 4k means only 25% of the total stream data. The scheme helps fully use the bandwidth and thus reduces the effect of asymmetric bandwidth, because any previously joined peers, including the previous joined leaf peers, can be selected to provide a patch stream. It also helps resist the network dynamics by using multiple data providers. 4.3 Analysis In this section, we briefly analyze the time complexity of the Nearcast overlay construction method. For simplicity, we assume that the average number of hosts in a subtree is m and all hosts are uniformly distributed in different subtrees. Based on the organization rules, we have H i=1 mi = N, where N is the total number of the joined hosts. Thus, m H N. Since the average height of a subtree is equal to log(m), we have the following inequality for the the height of the entire Nearcast tree: Height of tree H log(m) H log H N = log N. (2) For the host-join algorithm in Nearcast, in the worst case, the new host traverses the branch with the maximum height. Similarly, in the host-departure algorithm, the Probe message would also traverse the branch with the maximum height. Thus the time complexities of both the host-join and hostdeparture algorithms in Nearcast are on the order of O(log N). Suppose there are N peers in the system, among which v peers are fullfreeriders, and m half-freeriders come from n different NATs. Any open host can support about 1 d peers to join. We analyze the probability p s that any peer can successfully join the system even if there are many free-riders in the system. (See Section 2.1 for connectivity constraints due to free-riders.) We call two peers exclusive if they cannot join each other. In other words, they satisfy one of the following two conditions: (1) one peer is a half-freerider from a NAT, and another must be a full-freerider or a half-freerider from a different NAT; (2) both peers are full-freeriders. We randomly select two peers Y and X from N, and let X join Y. The rejection event happens only when the out-degree of Y runs out (given that Y is an open host) or Y and X are exclusive peers. The number of exclusive peers e is in the range m 1 + v < e < n 1 + v. Assuming the peers distribute uniformly in terms of connectivity constraints, we get p s 1 N v m Nd d i=1 ( ) e i+1 (v + m)e. (3) N N 2

7:12 X. Tu et al. The computation of p s is done as follows: (a) Y is an open host. The probability for a host to be an open host is N v m ; the N probability that Y can support i peers, 1 i d, to join the overlay is 1/d (i follows uniform distribution). Then the probability that i exclusive peers have already connected to Y is ( e N )i. Now the rejection can happen only when another peer X wants to join Y and X is an exclusive peer of Y (if X is not an exclusive peer, the Insert operation will be used to avoid rejection). The probability that X is an exclusive peer is e. All the above events happen N independently, so the probability that a rejection event happens when peer X joins Y which can support i connections is given by N v m ( e Nd N )i+1. Since 1 i d, we accumulate the probability for any peer i and get accumulation probability p i, p 1 = N v m d Nd i=1 ( e N )i+1 (b) Y is not an open host. The probability that Y is not an open host is v+m N. X is an exclusive peer of Y, so the probability p 2 that rejection happens is p 2 = v+m N ( e N ). Consequently, the probability that a peer can successfully join the overlay is given by p s 1 p 1 p 2 = 1 N v m Nd d i=1 ( ) e i+1 (v + m)e. N N 2 Using the log data shown in the last row of Table I (see Section 6.3), we estimate p s to be 95 96%, if d is set to 6. It proves that the system can support 95 96% peers to successfully join the overlay even if 21% of the peers are exclusive peers. 5. PERFORMANCE EVALUATION In this section, we report and discuss the simulation results for the performance evaluation of the proposed Nearcast approach, in comparison with an existing scheme called NICE 3 and the RTT approach which is simply used as a peer clustering metric. 5.1 Simulation Methodology We have implemented a packet-level simulator for simulating the three approaches. In the simulator, link queuing and packets losses are not considered because considering these network layer characteristics in the simulation is time-consuming and limits the scale of our simulation. The underlying network topologies used in our experiments are all generated by GT-ITM [GT-ITM 2000] in the Transit-Stub [Calvert et al. 1996] fashion. All the topologies include two transit domains and a total of 1000 routers with average out-degrees between 3 and 4. We assign link latencies of 150, 100, 50, and 20 ms to intertransits domain links, intratransit domain links, stub-transit links, and intrastub links, respectively. We also append 19,750 end-host nodes to stub routers randomly 3 We choose NICE for comparison because NICE outperforms ESM when the system scale is large, and it has the invariant for adapting with high-bandwidth applications.

Nearcast: Locality-Aware P2P Live Streaming Approach 7:13 and uniformly, as well as set the last hop latency to a random value between 2 to 5 ms. The upper bound of out-degrees for all the end-host nodes are randomly chosen from the range of 3 to 5. Thus, for NICE, the cluster size is bounded between 2 and 5 (i.e., k = 2). For the Nearcast approach, we assign four-dimensional network coordinates to end-host nodes by using the intrastub router number, intratransit stub number, intratransit router number, and transit number, respectively. Since NICE has no additional scheme to support hosts with connectivity constraints, for fairness of comparison, we assume that each end host is an open host. The experiment has two different phases: a join phase and a leave phase. In the join phase, a set of end-host nodes with the corresponding maximum group size join the overlay randomly in the first 200 s. The leave phase starts at time 1000 s, and thereafter all end-host nodes leave uniformly at random over duration of 200 s. 5.2 Performance Metrics Control overhead. This is measured by link stresses on the routers and hosts, where router stress and host stress are counted in the number of bytes traversed through a router and the number of control messages handled by an end host, respectively. Link stress. This is defined as the total number of identical copies of a data packet over a physical link. It accurately reflects the bandwidth efficiency of different approaches. End-to-end delay (EED). EED is the RTT between two neighboring hosts. It reflects the QoS issue with the use of a considerably large end system buffer. Absolute delay penalty (ADP). ADP 4 is defined as the latency for propagating a data packet along the overlay path from the source host to an end host. 5.3 Results 5.3.1 Control Overhead. From Figure 7 it can be seen that Nearcast carries much less control overhead compared with NICE and RTT. In NICE, group merging and division and periodically sending alive messages to each other for maintaining the group size lead to a large number of control messages and cause some stress on the routers. In RTT, there are many delay measurement messages injected into the overlay. In Nearcast, however, groups in each layer are organized by their locations, so there is little restriction on the group sizes. A peer s availability is detected by the interruption of the media data stream transmission. Therefore, the number of the control messages is greatly reduced. 5.3.2 Link Stress. From Figure 8(a), it can be found that, as the total number of nodes increases continuously, Nearcast has less average link stress than NICE, which slightly outperforms RTT. In NICE, the leader peer in each group knows only the distances between the peers within the group, but not the 4 We did not used the stretch metric because some recent studies [Hildrum et al. 2004] showed that stretch may not always be the case.

7:14 X. Tu et al. Fig. 7. Comparison of control overhead between Nearcast and NICE. Fig. 8. Comparison of link stress between Nearcast and NICE. distances between the peers outside the group. Thus, while the media data stream expands outward, members in an upper-layer group may not always choose the nearest peer as their supplier. Therefore, the average link stress increases. In Nearcast, however, intentional reservation according to the network location value is used to organize the subtree of each layer. This helps the supplier selection, and improves the efficiency of the physical network. As for the RTT scheme, three factors cause the higher link stress. First, the measurement of RTT can produce a lot of traffic. Second, like NICE, RTT does not consider the underlying physical topology. Third, RTT cannot accurately reflect the bandwidth of a link. In Figure 8(b), we plot a cumulative distribution of the stress for the entire set of 1024 peers at a time. Due to the inefficient use of links, the distribution of stress on links for NICE and RTT has a significantly larger tail than that of Nearcast. 5.3.3 End-to-End Delay and ADP. Figure 9(a) shows the distribution of the average EED and the average ADP for different group sizes. In all cases, Nearcast is better than NICE in terms of EED. This also is due to the inaccuracy in the supplier selection in NICE. The RTT scheme uses the RTT directly as the metric to cluster the near peers, and thus can perform better than the other two schemes in terms of EED. However, when the number of peers

Nearcast: Locality-Aware P2P Live Streaming Approach 7:15 Fig. 9. Comparison of end-to-end delay and absolute delay penalty between Nearcast and NICE. increases, the average link traffic increases so the EED becomes unstable and is usually different from the estimate of the first time. Consequently, the EED in RTT scheme becomes larger than that in Nearcast when the number of peers increases. As we can see in Figure 9(b), when there are a few peers in the multicast tree, the absolute latency in NICE and RTT is less than that in Nearcast. This is mainly because that Nearcast uses the peer s capability of reservation to organize the multicast tree, so that the capability of the upper-layered peers is not fully utilized, leading to the growth in height of the multicast tree. However, as the system scale grows gradually, the reserved capability of the upper-layered peers is utilized, which improves the absolute latency. 6. WIDE-AREA DEPLOYMENT OF NEARCAST We have deployed Nearcast in our live streaming broadcast system, which has been widely used in the China Education and Research Network (CERNET). In this section, we describe the performance of Nearcast in real-life experiments. 6.1 Deployment Considerations Two key issues should be carefully considered when deploying Nearcast for real use: (a) how to detect the connectivity constraints, and (b) how to generate the network coordinates before the end hosts join the overlay multicast tree. We discuss these issues in this section. 6.1.1 Detection of Connectivity Constraints. The connectivity constraint of an end host can be detected by the rendezvous point (RP) when it requests the IP address and port number of the source host. In particular, an half-freerider can be detected by comparing its public IP address and its private IP address (the private IP address should be included in the request message). For fullfreeriders, the RP uses the ping/pong mechanism to detect whether their serving ports are open. A full-freerider is identified when its serving port is not open. 6.1.2 Network Coordinates Generation. Usually, there are two types of methods for generating the network positions: static generation and dynamic

7:16 X. Tu et al. Fig. 10. The city-level backbone of CERNET. generation. In static generation, the mapping from the IP address to the coordinate of a peer is generated before the system starts to run. Dynamic generation means that the mapping can be generated at run-time when the peer joins the overlay. Dynamic generation methods are widely studied. Binning [Ratnasamy et al. 2002] can be used to generate multilayer single-dimension network coordinates, and the GNP method [Ng and Zhang 2002] provides a decentralized solution to generate the single-layer multidimensional coordinates of the end hosts. Nearcast employs a static method. Vertices in the layered physical network model have special geographical meanings. For instance, we separate the entire underlying Internet into a four-layer hierarchy, where vertices at layers 1, 2, 3, and 4 represent local-area networks (LANs), metropolitan-area networks (MANs), wide-area networks (WANs), and Internet service provider (ISPs) backbones, respectively. The geographical address information can be configured by a user or obtained from the free IP-to-geographical-address databases. 6.2 Deployment Environment The first version of our broadcast system was deployed on CERNET (China Education and Research Network), which covers more than 32 metropolises and 500 collages and institutions in China. Figure 10 shows the city-level backbone of CERNET. The entire system comprises four components: (1) an RP, (2) a media source, (3) a monitor, and (4) end systems. Figure 11 shows the system architecture of Nearcast deployment. Referring to this figure, each end system contains an IP-to-network coordinates database (INCD) which is prebuilt into

Nearcast: Locality-Aware P2P Live Streaming Approach 7:17 Fig. 11. The system architecture of Nearcast. the end system software. The value of the network coordinates of an end host is a 128-bit integer encoded by the four-layer geographical information corresponding to ISPs, cities, campuses, and buildings, respectively. Actually, the ISP-level coordinate value is not used since the current INCD just contains IP addresses in CERNET. The source host is responsible for receiving the encoded video streams and propagating them to the end hosts via the Nearcast approach. The end hosts initialize the join process by requesting the IP address and the port number of the source host and detecting their connectivity constraints with the help of the RP. Once an end host has received a video data packet, it puts the packet into a buffer of a size of 20 s packets, and sends it to the Windows Media Player (using ASF media format) or Winnamp (using NSV media format) for playback, as well as broadcasts the packet to its children. Meanwhile, the performance information is periodically reported to the monitor. Here, the reporting time interval is 2 min. The monitor is responsible for logging the performance information of the joined hosts, which includes the IP address, joining time, stay time, etc. The same information will be adopted when next the FCFS approach is used in the execution of the run procedure of the system. 6.3 Performance Analysis Our broadcast system has been used by HUSTOnline (http://www.hustonline. net) to broadcast TV streams with a 512kh/s bits rate to remote online students distributed to the sites of CERNET. We used the log traces gathered during the period from August 13, 2004 to August 29, 2004. During this period, the 2004 Athens Olympic Games were broadcast via our system. Since the performance data of the underlying network, such as router stress and link stress, is very difficult to gather, we analyzed the performance of Nearcast using the end host metrics: host stress, end-to-end delay, and absolute delay penalty. Logging traces showed that about 7200 users distributed over 100 colleges in 32 cities used our system for online learning. For ease of analysis, we use

7:18 X. Tu et al. Fig. 12. The single tree size versus time interval. Table I. Basic Statistical Information About Analyzed Trees; m/n in the Half-Freeriders Column Represents that m Half-Freeriders Come from n Different NATs Size Time Height Half-freeriders Full-freeriders 106 14/8/04 5 21/18 28 198 15/8/04 5 22/14 37 301 17/8/04 6 33/28 46 405 18/8/04 6 35/32 46 504 19/8/04 7 29/23 56 604 22/8/04 7 35/29 48 709 20/8/04 7 89/65 64 859 21/8/04 7 95/70 79 1038 24/8/04 7 108/76 103 1162 26/8/04 7 123/85 120 10 min as a unit to reconstruct the Nearcast tree based on the logging traces. Figure 12 shows the changes in the size of a single tree over 16 days. It can be seen that the maximum number of users severed by a single Nearcast tree was 1162. We analyzed the system performance in detail with the selected 10 trees which were generated at different times and of different sizes between 100 and 1200 nodes. Table I shows the basic statistics on the size, time, and height of a tree, number of half-freeriders, and full-freeriders. From Figure 13 we can see that the end-to-end latencies of most parent-to-child pairs are less than 100 ms, which can be attributed to the locality awareness. This effect is also shown in Figure 14(a). Figure 14(a) shows the difference in the cumulative distribution function of the end-to-end delay between the Nearcast and the FCFS methods, with the overlay tree size of 1162 users. In the figure, for locality-aware multicast, the end-to-end delays for almost 95% parent-to-child pairs are less than 100 ms. This result is also depicted in Figure 13. However, for the FCFS method, almost

Nearcast: Locality-Aware P2P Live Streaming Approach 7:19 Fig. 13. Distribution of average EED. Fig. 14. Comparison of average EED between Nearcast and FCFS. 93% of the end-to-end delay values are within the range [100 ms, 300 ms]. Figure 14(b) compares the average end-to-end delay of the Nearcast method and the FCFS method. We can see from the figure that the average end-to-end delay in Nearcast is far less than that in the randomly constructed tree with the FCFS method. As a matter of fact, over 90% of Nearcast users have reported (either by emails to us or through BBS feedback) that the video stream can be played back smoothly without any jitter. As shown in Table I, the heights of the locality-aware multicast trees with the sizes of 106 to 1162 users are larger than or equal to 5. Obviously, the

7:20 X. Tu et al. Fig. 15. Comparison of average ADP between Nearcast and FCFS. height of the FCFS tree is less than log 5 N, where 5 is the out-degree of an end host and N is the number of the joined hosts. Thus, for FCFS trees with a size less than 1162, the corresponding height is also less than log 5 1162 5. Clearly, the height of the locality-aware overlay multicast tree is large because certain out-degrees of some hosts are reserved to serve future nearer hosts. However, as shown in Figure 15, the absolute delay penalty of the localityaware overlay tree is far less than that of the FCFS tree. In Figure 15(a), with a tree of the size of 1162 users, the absolute delay penalties for 93% of the users are between 100 and 400 ms, while the absolute delay penalties for most users in the FCFS tree are between 200 and 600 ms. Figure 15(b) shows a clear difference of the average absolute delay penalties between the Nearcast tree and the FCFS tree. From this figure, we can see that the average absolute delay penalty for the FCFS method is three times that for the locality-aware construction method when the tree size is larger than 1000 users. Figure 16 shows the minimum, maximum, and average ADP in Nearcast. It can be concluded that the average ADP is very low, and close to the minimum ADP, which means that very few peers have a high ADP. The use of the network address translator (NAT) partially solves the address exhaustion problem of IPv4 and the firewall provides a solution to address the security issue. However, they raise many challenges for peer-to-peer applications since the hosts behind the NAT gateways or firewalls are often restricted to serve receivers only, not suppliers. As observed from several live Internet video broadcasts, the fraction of the NAT and the firewall hosts in a multicast group can be as high as 80% [Chu et al. 2004]. Ganjam and Zhang [2004] also described the experience of deploying a video broadcast application using an overlay multicast tree. As our log indicates (shown in Table I), nearly 20% 45% of the Nearcast users are behind NATs or firewalls. In our implementation, we have used the inserting mechanism to make the hosts behind the NAT or the firewalls as leaves of the overlay multicast tree. Based on the nearest-parent selection principle, users behind the same NAT gateway can be organized into the same subtree. It extends the service capacity of the hosts behind the NAT.

Nearcast: Locality-Aware P2P Live Streaming Approach 7:21 Fig. 16. Cumulative distribution of ADP versus tree size. As shown in Table I, in the real-world network environment, nearly 10% of the NAT users are behind the same NAT gateway. 7. CONCLUSIONS In this article, we have presented Nearcast, a locality-aware P2P live video streaming approach to deliver video courses to a large number of remote students in distance education applications. Nearcast is a single-source overlay multicast approach that uses prebuilt network coordinates of the end hosts to organize the overlay multicast tree. It takes into consideration the topology mismatch problem and considers the network constraints, such as limited last-hop bandwidth and connectivity constraints in the tree construction and recovery procedures. We have evaluated the performance of Nearcast using both simulations and real deployment experiments. Compared to the existing approaches, Nearcast significantly improves the system performance with lower link stress and network latency. REFERENCES ANDERSEN, D., BALAKRISHNAN, H., FRANS KAASHOEK, M., AND MORRIS, R. 2001. Resilient overlay networks. In Proceedings of the 18th ACM Symposium on Operating Systems Principles. 131 145. BANERJEE, S., BHATTACHARJEE, B.,AND KOMMAREDDY, C. 2002. Scalable application layer multicast. In Proceedings of the ACM SIGCOMM. 205 217. BRITO, G. A. A. D. AND MOURA, A. M. de C. 2001. ROSA-P2P: A peer-to-peer system for learning objects integration on the Web. In Proceedings of the 11th Brazilian Symposium on Multimedia and the Web. 1 9. CALVERT, K., ZEGURA, E., AND BHATTACHARJEE, S. 1996. How to model an internetwork. In Proceedings of IEEE INFOCOM. 594 602.

7:22 X. Tu et al. CASTRO, M., DRUSCHEL, P., KERMARREC, A. M., NANDI, A., ROWSTRON, A., AND SINGH, A. 2003. Split- Stream: High-bandwidth multicast in a cooperative environment, In Proceedings of the Symposium on Operating Systems Principles (Lake Bolton, NY). 298 313. CHAWATHE, Y. 2000. Scattercast: An architecture for Internet broadcast distribution as an infrastructure service. Ph.D. dissertation. University of California, Berkeley. CHU, Y.-H., RAO, S. G., AND ZHANG, H. 2001. Enabling conferencing applications on the Internet using an overlay multicast architecture. In Proceedings of ACM SIGCOMM. 55 67. CHU, Y.-H., RAO, S. G., AND ZHANG, H. 2004. Early deployment experience with an overlay based Internet broadcasting system. In Proceedings of USENIX Annual Technical Conference. DEERING, S. AND CHERITON, D. 1990. Multicast routing in datagram internetworks and extended LANs. In ACM Trans. Comput. Syst. 8, 2 (May), 85 100. GT-ITM 2000. Website: http://www.cc.gatech.edu/projects/gtitm/. GANJAM, A.AND ZHANG, H. 2004. Connectivity restrictions in overlay multicast. In Proceedings of NOSSDAV. 54 59. GUYTON, J. D. AND SCHWARTZ, M. F. 1995. Locating nearby copies of replicated Internet servers. In Proceedings of ACM SIGCOMM. 288 298. HEFEEDA, M., HABIB, A., BOTEV, B., XU, D., AND BHARGAVA, B. 2003. PROMISE: Peer to peer media streaming using CollectCast. In Proceedings of ACM Multimedia (Berkeley, CA). 45 54. HILDRUM, K., KUBIATOWICZ, J. D., AND STRIBLING, J. 2004. Brief announcement: Stretch between nearby peers. In Proceedings of the 23rd Annual ACM Symposium on Principles of Distributed Computing (St. John s, Newfoundland, Canada). 367 367. JANNOTTI, J., GIFFORD, D., JOHNSON, K., KAASHOEK, M., AND O TOOLE, J. 2000. Overcast: Reliable multicasting with an overlay network. In Proceedings of the 4th Symposium on Operating Systems Design and Implementation. 197 212. JIN, H.YIN, Z., YANG, X., WANG, F., MA, J., WANG, H., AND YIN, J. 2004. APPLE: A novel P2P based e-learning environment. In Proceedings of the 6th International Workshop on Distributed Computing. (Indian Statistical Institute, Kolkata, India). (Website: http://grid.hust.edu.cn/apple/). KOSTIC, D., RODRIGUEZ, A., ALBRECHT, J., AND VAHDAT, A. 2003. Bullet: High bandwidth data dissemination using an overlay mesh. In Proceedings of SOSP (Bolton Landing, NY). 282 297. KWON, M. AND FAHMY, S. 2002. Topology-aware overlay networks for group communication. In Proceedings of ACM NOSSDAV. 127 136. LIU, L. S. AND ZIMMERMANN, R. 2006. Adaptive low-latency peer-to-peer streaming and its application. In Multimedia Systems. Springer, New York, NY, 497 512. LIANG, J. AND NAHRSTEDT, K., 2006. DagStream: Locaity aware and failure resilient peer-to-peer streaming. In Proceedings of SPIE/ACM MMCN. LIAO, X., JIN. H., LIU, Y., NI. L.M.,AND DENG, D. 2006. AnySee: Peer to peer live streaming. In Proceedings of IEEE INFOCOM. LIU, Y., LIU, X., XIAO, L., NI, L. M., AND ZHANG, X. 2004. Location-aware topology matching in P2P systems. In Proceedings of IEEE INFOCOM. (Hong Kong, China). 2220 2230. NG, T. S. E. AND ZHANG, H. 2002. Predicting Internet network distance with coordinates-based approaches. In Proceedings of IEEE INFOCOM. 170 179. PADMANABHAN, V. N., WANG, H. J.,CHOU, P. A., AND SRIPANIDKULCHAI, K. 2002. Distributing streaming media content using cooperative networking. In Proceedings of ACM NOSSDAV. 177 186. PENDARAKIS, D., SHI, S., VERMA, D., AND WALDVOGEL, M. 2001. ALMI: An application level multicast infrastructure. In Proceedings of the 3rd Usenix Symposium on Internet Technologies & Systems. RATNASAMY, S., HANDLEY, M., KARP, R., AND SHENKER, S. 2001. Application-level multicast using content-addressable networks. In Proceedings of the 3rd International Workshop on Networked Group Communication. 14 21. RATNASAMY, S., HANDLEY, M., KARP, R., AND SHENKER, S. 2002. Topologically-aware overlay construction and server selection. In Proceedings of IEEE INFOCOM (New York, NY). 1190 1199. ROWSTRON, A. AND DRUSCHEL, P. 2001. Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In Proceedings of IFIP/ACM International Conference on Distributed Systems Platforms (Middleware).