Mutualcast: An Efficient Mechanism for One-To-Many Content Distribution ABSTRACT Categories and Subject Descriptors General Terms Keywords

Transcription

1 Mtalcast: An Efficient Mechanism for One-To-Many Content Distribtion Jin Li, Philip A. Cho and Cha Zhang, Microsoft Research, Commnication and Collaboration Systems Grop One Microsoft Way, Bld. 3, Redmond, WA {jinl, pacho, ABSTRACT In this paper, we propose Mtalcast, a new delivery mechanism for content distribtion in peer-to-peer (P2P) networks. Compared with prior one-to-many content distribtion approaches, Mtalcast achieves fll tilization of the pload bandwidths of the peer nodes, thereby maximizing the delivery throghpt. Mtalcast splits the to-be-distribted content into many small blocks, so that the more resorcefl nodes may redistribte more blocks, and the less resorcefl nodes may redistribte fewer blocks. Each content block is assigned to a single node for distribtion, which can be a content-reqesting peer node, a non-content-reqesting peer node, or even the sorce node. The throghpt of the distribtion is controlled by redistribtion qees between the sorce and the peer nodes. Frthermore, Mtalcast can be reliable and synchronos. Ths, it can be applied to file/software downloading, media streaming, real-time adio/video conferencing, etc. Categories and Sbject Descriptors C.2.4 [Distribted Systems]: Distribted Applications; C.2.5 [Local and Wide-Area etworks]: Internet. General Terms Algorithms, Design, Performance Keywords One-to-many content distribtion, file distribtion, software distribtion, peer-to-peer networks. Permission to make digital or hard copies of all or part of this work for personal or classroom se is granted withot fee provided that copies are not made or distribted for profit or commercial advantage and that copies bear this notice and the fll citation on the first page. To copy otherwise, or repblish, to post on servers or to redistribte to lists, reqires prior specific permission and/or a fee. ACM Sigcomm Asia 04, Apr. 0 2, 2004, Beijing, China. Copyright 2004 ACM ITRODUCTIO A nmber of applications sch as software distribtion, Internet TV/video streaming, video conferencing, mltiplayer gaming, personal media distribtion, and P2P web content dplication distribte content from one sorce node to many destination nodes (one-to-many content distribtion). A network-level soltion to efficiently distribte the content is IP Mlticast [], where a single packet transmitted from the sorce is dplicated at roters along a distribtion tree rooted at the sorce node, and is thereby delivered to an arbitrary nmber of receivers. Thogh IP mlticast is an efficient soltion, its deployment is slow in the real world becase of isses sch as inter-domain roting protocols, ISP bsiness models (charging models), congestion control along the distribtion tree, secrity and so forth. As a reslt, the vast majority of traffic in the Internet today is nicast based, where two compters directly talk to each other. Since IP mlticast is not generally feasible, varios approaches have been developed to let peer compters, instead of roters, distribte the content from the sorce. The general approach is application-level mlticast (ALM) [4], where a mlticast distribtion tree is formed and overlaid on the existing network. Instead of sing IP mlticast, each peer compter in the distribtion tree implements all mlticast related fnctionalities inclding packet replication, membership management and content delivery on the overlaid network. Some sample systems inclde Scattercast [2] and Overcast [3], both of which se a single tree to distribte the content. Compared with letting the sorce node directly send its content to all other clients, the distribtion tree approach redces the network load of the sorce, ths achieving more efficient content distribtion. In a distribtion tree, the intermediate nodes redistribte the content, while the leaf nodes only receive the content. The pload bandwidths of the leaf nodes are not tilized for content distribtion. Coopet [5] and SplitStream [6] overcome this inefficiency by splitting the content into mltiple stripes and distribting the stripes across separate mlticast trees with disjoint interior nodes. Any peer compter cold be an interior node in one of the mlticast trees, and contribte to forwarding the content. FastReplica [7] and Bllet [8] investigate the isse of efficient and reliable replication of large files. If there are n nodes, FastReplica first partitions the file into n sbfiles of eqal size. Each sbfile is then transferred to a different peer in the grop, which is sbseqently replicated and transferred to the other peers. In Bllet, peer nodes are organized into an overlay tree. Each node splits the content received from the parent into a disjoint set of blocks, with each set sent to a different child node. The child nodes then discover the missing blocks and the nodes that hold the missing blocks, and send reqests to recover the missing blocks. A related scheme, sing erasre coded blocks, is proposed in [9]. A practical P2P system has been implemented by BitTorrent [20] with sharing incentive so that each pair of peers roghly sends and receives an eqal amont of content. These are jst a few examples of the many recent schemes for application-level mlticast. Althogh the above ALM distribtion strategies are more efficient than directly sending content from the sorce to the peers, they fail to achieve maximally efficient content distribtion in the network. one of the above schemes adeqately considers the differences in bandwidth between the peer nodes. They also fail to flly engage the bandwidth resorces of all the peer nodes to distribte the content. Frthermore, the above ALM distribtion strategies are either asynchronos or nreliable. Systems sch as Coopet and Split- Stream, which are sed for synchronos delivery of content (e.g., streaming adio or video), while resilient, are nreliable in the sense that they cannot garantee delivery of every byte. Con-

2 Figre One-to-many content distribtion. versely, systems sch as FastReplica, Bllet, and BitTorrent, which are sed for reliable file distribtion, are asynchronos in the sense that content may arrive at each receiver at a different time and at a different rate. Althogh there are prior mlticast soltions that are both reliable and synchronos (as typified by SRM [2], RMTP [22], and the like [23]), in these prior soltions reliability and synchrony are achieved, ltimately, by slowing down the transmission to the rate of the lowest bandwidth link, which is a very inefficient soltion in a heterogeneos environment. In this work, we propose Mtalcast, which is a maximally bandwidth-efficient mechanism for content distribtion in the P2P network, and which frthermore is synchronos and can be reliable. In contrast to existing approaches, Mtalcast achieves the maximm possible throghpt for its content by ) engaging as many nodes as possible to distribte the content and 2) flly tilizing the nodes' available pload bandwidths. Mtalcast also adjsts the content sending rate dynamically to match the maximm throghpt nder the prevailing network conditions. Mtalcast is simple and flexible, with three distinct featres. First, it splits the to-be-distribted content, be it a file or media stream, into many small blocks, and distribtes each block separately. The blocks are pipelined for delivery, allowing for overall synchronos commnication. Second, in a Mtalcast grop, each block of content is assigned to a single peer node for redelivery. The assignment and redelivery can be done reliably, allowing for overall reliable commnication. Third and most importantly, Mtalcast employs an optimal bandwidth allocation strategy, which is implemented via redistribtion qees between the sorce and the peer nodes. The redistribtion qees accommodate the bandwidth differences between the peer nodes, and ensre maximm delivery throghpt even if there are packet loss and transmission jitter on network links. This ensres maximm efficiency. We have implemented Mtalcast for file/software distribtion, e.g., distribtion of software patches/pdate. evertheless, the Mtalcast protocol can be sed to distribte other content, sch as streaming media or mltimedia collections. 2. BACKGROUD AD PRIOR WORKS The one-to-many content distribtion problem that we are considering is illstrated in Figre. The network consists of a sorce node s, which holds the content to be distribted, and mltiple peer nodes t i, i =,2,,, each of which may or may not reqest a copy of the content. The nmber of peer nodes is on the order of tens. Both the sorce node and the peer nodes are compters connected to the Internet throgh an internet service provider (ISP), sing an ADSL, cable modem, camps, or corporate network link. We do not consider the case that the nodes are infrastrctre nodes, sch as the roters, on the backbone of the Internet. Or target is to distribte the content with maximm throghpt to all the destinations. The simplest approach for a sorce node to distribte content in this setting is to let the sorce node send the content directly to the destination nodes. Thogh straightforward, the throghpt of the content distribtion is bonded by the pload bandwidth of the sorce node, which is sally fairly limited. atrally, we want to enlist the help of the peer nodes, and se their pload bandwidths to aid the content distribtion. Application-level mlticast (ALM) [4] has attracted a lot of interest recently. A few ALM systems, sch as ICE [8], Scribe [9], Bayex [0] and CA-mlticast [], are designed for large-scale applications. Among them, ICE has its own node management mechanism, while Scribe, Bayex and CA-mlticast are implemented on strctred peer-to-peer overlay networks sch as Pastry [7], Tapestry [2] and CA [3]. The main objective of these large-scale systems is to redce/balance the link and node stress while limiting the delay and dplication [6]. On the other hand, when the system scale is relatively small (as shown in Figre ), maintaining the grop stats is relatively simple. The focs is ths to manage the mlticast roting sch that the system is reliable, efficient and has low end-to-end delay. We next examine a few prior small-scale ALM approaches that are directly related to or Mtalcast framework. Scattercast [2] and Overcast [3] each form a single distribtion tree. A sample distribtion tree is shown in Figre 2(a). In this configration, the sorce node sends the data to node t, which forwards the data to nodes t 2 and t 3. The ALM distribtion tree tilizes the pload bandwidth of the intermediate node t, whereas the pload bandwidths of the leaf nodes t 2 and t 3 are not tilized. To better tilize the bandwidths of the peer nodes, nodes with higher pload bandwidths shold be placed pstream, while nodes with lower pload bandwidths shold be placed downstream. Both Coopet [5] and SplitStream [6] stripe the content and distribte the stripes sing separate mlticast trees with disjoint interior nodes. Coopet ses a centralized tree management scheme, while SplitStream relies on Pastry [7] to maintain the distribtion tree. Coopet frther tilizes mltiple description coding (MDC) and forward error correction (FEC) to protect from packet loss and node failre. The Coopet/SplitStream configration with two application-level mlticast trees is illstrated in Figre 2(b). The content is divided into two eqal stripes. The first stripe is sent to node t, which forwards the stripe to nodes t 2 and t 3. The second stripe is sent to node t 2, which forwards the strip to nodes t and t 3. We notice that the system tilizes the pload bandwidths of nodes t and t 2, bt fails to tilize the pload bandwidth of node t 3. The systems are resilient bt not reliable. FastReplica [7] is specifically designed for file download. For an node P2P network, FastReplica distribtes the file with height-2 mlticast trees with intermediate degree -. A sample FastReplica configration of three peer nodes is illstrated in Figre 2(c). FastReplica distribtes the file in two steps: the distribtion step and the collection step. In the distribtion step, the file is split into three sbfiles and sent to nodes t, t 2 and t 3 (along solid, dashed, and dotted lines), respectively. After the distribtion step, the collection step kicks in. Each peer node forwards its sbfile to the other peer nodes. All peer nodes are engaged in content distribtion in FastReplica. FastReplica is reliable, bt

3 t 2 t t t s s t 3 s t 2 t 3 t 2 t 3 (a) (b) (c) Figre 2 Prior one-to-many content distribtion schemes: a) Scattercast/Overcast, b) SplitStream/Coopet, c) FastReplica. becase the distribtion is split into two stages, it cannot be applied to synchronos applications sch as streaming media. All of the above one-to-many content distribtion approaches adapt to the capabilities of the peer nodes, i.e., their pload/download bandwidths, by establishing a sitable network topology. odes with high bandwidth are placed in the center of the distribtion network, and are in charge of more content distribtion. Once the network topology is established, it is often fixed throghot the session. Sch distribtion strategy leaves the distribtion network less flexible to adapt to changes in the network conditions, e.g., congestion of certain nodes/links. An alternative approach is to constantly pdate the network topology by measring the network characteristics dring the session. For instance, in End System Mlticast [4], a protocol called arada is developed to constrct an overlay strctre among participating end systems in a self-organizing and flly-distribted manner. End systems gather information of network path characteristics sing passive monitoring and active measrements. arada continally refines the overlay strctre as more network information is available. ALMI [5] is another protocol that can adapt its strctre to the network condition. Each ALMI session has a session controller which constrcts a shared tree. The tree is periodically re-calclated based on the end-to-end measrements collected by session members. Adapting the network topology to network conditions makes arada and ALMI robst to network condition variations. Unfortnately, both arada and ALMI may sffer from inefficiency or trblence dring the adjstment of the network topology. 3. MUTUALCAST DISTRIBUTIO 3. Framework Mtalcast differs from the previos one-to-many content distribtion approaches in that it ses a fixed network topology, bt adapts by letting peer nodes with different capabilities distribte different amont of content. Mtalcast is simple to implement, with several distinct featres. First, Mtalcast splits the to-bedistribted content, be it a file or a media stream, into many small blocks. The nmber of blocks redistribted by a certain node can ths be proportional to the resorce (pload bandwidth) of the node. The node with larger pload bandwidth may redistribte more blocks, and the node with smaller pload bandwidth may redistribte fewer blocks. Second, in a Mtalcast grop, each block of content is assigned to a single node for redelivery. The node in charge of the redelivery can be a content-reqesting peer node, a non-content-reqesting peer node, or even the sorce node itself. Third, employing redistribtion qees between the nodes, Mtalcast can effectively deal with dynamic changes in the network condition, and copes with variations in the pload Figre 3 Mtalcast content distribtion network: an example. bandwidth, packet loss and packet jitter on an ongoing basis. An additional featre is that the network topology of Mtalcast is generic and fixed. Mtalcast relies on bandwidth reallocation to adapt to the network condition variations, which is preferred to adapting the network topology itself. The basic distribtion framework of Mtalcast is as follows. The content being distribted is chopped into blocks B j, j=,2,,m. For each block B j, one niqe node is selected to distribte the block to the rest of the peer nodes. Freqently, the node in charge of redistribting the block B j is a peer node t i. In sch a case, the sorce node sends one copy of the block B j to the peer node t i, which then redistribtes the block B j by sending a copy of the block to the rest of the peer nodes. However, when the sorce node has abndant bandwidth resorces, the node in charge of distribting the block B j can be the sorce node s itself. In that case, the sorce node will directly send one copy of block B j to each peer node t i. We show an example Mtalcast distribtion network in Figre 3. In this network, there are one sorce node s and for peer nodes t, t 2, t 3 and t 4. Among the peer nodes, the nodes t, t 2 and t 3 reqest a copy of the content from the sorce node s. The node t 4 does not reqest a copy of the content. evertheless, it contribtes its pload bandwidth to help distribting the content to the other peer nodes. When the block is assigned to the content-receiving peer nodes t, t 2 and t 3 for redistribtion, sch as the blocks, 2, 3 and 4, the block is first sent by the sorce node to the peer node in charge, which then forwards the block to the other two peer nodes. When the block is assigned to a non-content-receiving peer node t 4 for redistribtion, sch as the blocks 5, 6 and 7, the block is first sent by the sorce node to the peer node t 4, which forwards the block to the other three peer nodes. The sorce node may also choose to directly distribte the block, sch as the block 8. In that case, the block is sent directly from the sorce node to the peer nodes t, t 2 and t 3. Mtalcast chops the content into a large nmber of small blocks for distribtion. The size of the Mtalcast block is a compromise between the granlarity of distribtion and the overhead reqired for identifying the block. Dring the implementation, it is preferable that the size of the Mtalcast block is a little bit less than the maximm transmission nit (MTU) of the network, so that each Mtalcast block can be sent as a single packet over the network. In the crrent work, we set the block size as KB.

4 3.2 Mtalcast: distribtion rotes Mtalcast assigns each block to a certain node for redistribtion. The nmber of blocks assigned to the peer node is proportional to its capacity, which is evalated by its pload bandwidth. The reason is the following. In terms of the contribtion of a peer node to the network, it is the pload bandwidth of the peer node that conts. Ths, to efficiently distribte content in a Mtalcast network, we shold make se of the pload bandwidths of the peer nodes as mch as possible. We also notice that for a file distribtion session, the primary parameter that governs the speed of the distribtion is the throghpt of the network link. If a client can choose mltiple servers to serve it the file, it shold choose the server that provides the fastest network throghpt between the two. The other network parameters, sch as rond trip time (RTT), packet loss ratio, network jitter, are less relevant. In a networks composed of the end-ser nodes, we may characterize the network by assigning an pload bandwidth constraint on each node, a download bandwidth constraint on each node, and a link bandwidth constraint between any two nodes or any two grop of nodes. However, the bottleneck is sally the pload bandwidths of the nodes. This is becase in Mtalcast, a peer node sends content to mltiple destinations. The otpt of the peer node ths splits among mltiple receivers. As a reslt, the link bandwidth reqired between the two peer nodes is only a fraction of the pload bandwidth of the sending node, which sally does not become the bottleneck. The reqired download bandwidth for a node to receive the content is always less than the total available pload bandwidths of all the nodes in the network divided by the total nmber of receiving nodes. In increasingly common networks, sch as cable modem and ADSL networks, the total pload bandwidths of the end-ser nodes are mch smaller than the total download bandwidths. Even for ser nodes on the camps networks or the corporate networks, the download bandwidth can still be mch larger than the available pload bandwidth becase the ser may cap the pload bandwidth to limit participation in the P2P activity. In the following discssion, we will assme that the receiving nodes have enogh download and link bandwidths to receive content from the Mtalcast. We will briefly discss nodes of limiting download and link bandwidths in Section 3.7. Table Link bandwidth and download bandwidth reqirement for Mtalcast network of Figre 3. Receiving Sending node, and Link Bandwidths Download node s t t 2 t 3 t 4 Bandwidth t 0.83B - 0.5B B B 3.33B t B 0.5B - B B 3.33B t 3.33B 0.5B 0.5B - B 3.33B t 4 B B Upload BW 4B B B 2B 3B In the example of Figre 3, let s assme that the pload bandwidths of the peer nodes t and t 2 are B; that of the peer node t 3 is 2B; that of the peer node t 4 is 3B; and that of the sorce node is 4B, where B is a nit of bandwidth. An optimal strategy of flly tilizing the pload bandwidths of the sorce and peer nodes is shown in Table. We will discss the bandwidth allocation problem in the next section. If the Mtalcast grop incldes a sorce node, contentreqesting peer nodes ( > as otherwise the problem is trivial) and 2 non-content-reqesting (bt willing to participate) peer nodes, the Mtalcast network will distribte the content throgh height-2 trees with intermediate degree - (with the intermediate node being one of the content-reqesting nodes), 2 height-2 trees with intermediate degree (with the intermediate node being one of the non-content-reqesting nodes), and one height- tree with degree, all rooted at the sorce node. The network topology employed by Mtalcast bears some resemblance to the FastReplica scheme of [7]. evertheless, there are a nmber of distinct featres of Mtalcast. First, Mtalcast does not separate the distribtion and the collection steps. Instead, the content blocks are distribted continosly by the sorce and the peer nodes. Second, in Mtalcast, the amont of content being redistribted by a particlar peer is not fixed, bt varies according to the capabilities (the pload bandwidths) of the peer nodes. Finally, Mtalcast may involve the sorce node and non-contentreqesting peer nodes in the redistribtion of content. The Mtalcast network distribtes the content throgh three rotes: ) throgh content-reqesting peer nodes, 2) throgh noncontent-reqesting peer nodes, and 3) directly from the sorce node. Each distribtion method demands different amonts of network resorce from the participating nodes. Again, the network resorce of chief concern is the pload bandwidth consmed. To distribte a portion of content having bandwidth B in a Mtalcast network of content-reqesting peer nodes, the first distribtion rote demands pload bandwidth B from the sorce node, and pload bandwidth ( -)B from each content-reqesting peer node. The second distribtion rote demands pload bandwidth B from the sorce node, and pload bandwidth B from each noncontent-reqesting peer node. The third distribtion rote demands pload bandwidth B from the sorce node. In the first and the second rotes, Mtalcast ses the pload bandwidths of the peer nodes (inclding the content-reqesting peer nodes and the non-content-reqesting peer nodes) to alleviate the pload bandwidth brden on the sorce node. This has the effect of speeding p the maximm rate of content distribtion. It is interesting to notice that for the same rote, the amont of network resorce consmed is independent of the individal pload bandwidth of each peer node. Ths we may consider the bandwidth allocation problem with respect to each rote category instead of each peer node. 3.3 Mtalcast: bandwidth allocation In the Mtalcast network, the most precios resorce is the pload bandwidth of the sorce node, where the content originates. If the pload bandwidth of the sorce node is sed p, we cannot frther speed p content distribtion, even if there are still peer nodes with available pload bandwidths. It is apparent that if the sorce node sends content blocks at rate B throgh the delivery links to all content-reqesting peer nodes, it will consme B of the pload bandwidth of the sorce. On the other hand, if the sorce node sends content blocks at rate B to a peer node t i, which in trn distribtes the blocks to the rest of the content-reqesting peer nodes, only an amont B of the pload bandwidth of the sorce node is needed. Apparently, as long as there are more than one content-reqesting peer nodes, the sorce node shold forward as many content blocks as possible to the peer nodes for redelivery. Between the content-reqesting and non-contentreqesting peer nodes, the content-reqesting peer nodes have a slight edge in efficiency, as the content blocks sent to the nodes in

5 the forward links are not wasted. As a reslt, among the three distribtion rotes otlined above, the most preferred rote is rote, followed by the rote 2. Only when the sorce node still has pload bandwidth left, it may choose rote 3 to distribte content directly to the peer nodes. We assme that the Mtalcast network consists of a sorce node of pload bandwidth B s, ( >) content-reqesting peer nodes with average bandwidth B, and 2 non-content-reqesting peer nodes with average bandwidth B 2. Applying the distribtion rote selection strategy above, the distribtion throghpt of the Mtalcast network, which is defined as the amont of content sent to the content-reqesting peer nodes per second is: Bs Bs Bs + Bs 2, θ = Bs ( Bs + Bs 2 ) ( Bs + Bs 2 ) + Bs Bs + Bs 2, () with B s = B and B s 2 2 = B2. Figre 4 The forward link thread of the peer node. This shows that before the pload bandwidths of all the peer nodes have been exhasted, the distribtion throghpt is limited only by the pload bandwidth of the sorce node. All contentreqesting peer nodes receive content at the rate of the pload bandwidth of the sorce node. After the pload bandwidths of all the peer nodes have been exhasted, the distribtion throghpt becomes (/ ) th of the sm of the pload bandwidths of the network ( B + 2 B 2 + B s ) mins a small portion ( 2 B 2 / ) wasted in the distribtion throgh non-content-reqesting peer nodes. 3.4 Mtalcast: distribtion rote selection throgh redistribtion qee With the priority otlined in Section 3.3, if we know the available pload bandwidths of the sorce and all the peer nodes, we may explicitly calclate the bandwidth allocated between any two peer nodes, and distribte content blocks accordingly. However, there is an even simpler method that works in a distribted fashion. We may se a qee to estimate the bandwidth on any connection link, and govern the selection of the distribtion rotes of the content blocks based on the stats of the qees, ths achieving implicit bandwidth allocation withot knowing the bandwidths of the network. The key idea is to establish a qee to bffer content being sent from one node to another, and to se the qee to control the speed of distribtion between any two nodes. In or implementation of Mtalcast, the links between nodes are established via TCP connections. The redistribtion qees are ths simply the TCP send and receive bffers. An additional advantage of sing TCP is that the flow control, reliable data delivery and node leave events are all atomatically handled by TCP. Reliable data delivery in Mtalcast is inherited throgh these reliable TCP connections. Congestion control in Mtalcast is likewise inherited. We call the TCP connection carrying blocks to be redistribted the forward link, and the TCP connection that carries blocks not to be frther redistribted the delivery link. We establish one TCP connection (the delivery link) from each peer node to every other content-reqesting peer node. We establish one TCP connection (the forward link) from the sorce node to every non-contentreqesting peer node, and two TCP connections (the forward and Figre 5 The delivery link thread of the peer node. the delivery links) from the sorce node to every contentreqesting peer nodes. The selection of the distribtion rotes becomes finding available slots in the TCP connections. Let s now examine the workflow of the Mtalcast sorce and peer nodes. Each content-reqesting Mtalcast peer node consists of two threads, where one thread receives the content blocks from the delivery link, while the other thread receives the content blocks from the forward link and redistribtes them to the rest of the content-reqesting peer nodes throgh their delivery links. For the non-content-reqesting peer nodes, only the forward link thread is operated. The operational flow of the forward link thread of a Mtalcast peer node (both content-reqesting and non-content-reqesting) is shown in Figre 4. In each iteration loop of the forward link thread, the peer node removes one content block from the incoming forward link, and copies the block onto the otgoing delivery links of all the other content-reqesting peer nodes. The thread does not remove another content block from the incoming forward link ntil it has sccessflly copied the last content block onto all the delivery links. That way, if the otgoing delivery links are blocked, possibly reslted from reaching the limit on the pload bandwidth of the peer node, the peer node will stop removing the content blocks from the incoming forward link, ths effectively reglate the receiving rate of the forward link to be /M th of the pload bandwidth of the peer node, where M is the nmber of nodes that the content block is redistribted to, which is - for content-reqesting peer node and for non-content-reqesting peer node.

6 The operational flow of the delivery link thread of the contentreceiving peer node is shown in Figre 5. For the content blocks arriving on delivery links from nodes other than the sorce node, the operation is simply to remove the content blocks from the link as soon as they arrive. For content blocks arriving on the delivery link from the sorce node, we pt an additional constraint that we only remove content blocks from the delivery link when the receiving bffer length of the forward link from the same sorce node is above a certain threshold. The rationale is that the delivery link and the forward link are two separate TCP connections sharing the same network path from the sorce to the peer node. The content blocks sent throgh the forward link have higher priority, as they are to be redelivered to the other content receiving peers. The receiving bffer length policy garantees that the bandwidth of the forward link to be at least /M th of the pload bandwidth before the delivery link from the sorce node to the peer node is activated. The operational flow of a Mtalcast sorce node is shown in Figre 6. For each content block, the sorce node selects one of the distribtion rotes based on the stats of the redistribtion qee. The rote selection is based on the following order of priorities. The redistribtion by a content-reqesting peer node has the highest priority. The redistribtion by a non-contentreqesting peer node has the second highest priority. The distribtion directly from the sorce node to all the content-reqesting peer nodes has the lowest priority. As shown in Figre 6, the sorce node first checks if there is space available for the content block in any TCP connection of the forward link from the sorce node to the content-reqesting peer node. If the send bffer of one of the TCP connections is not fll, the content block is pt into that TCP bffer to be sent to the corresponding content-reqesting peer node, which then redistribtes the content block to the other content-reqesting peer nodes throgh the corresponding delivery links. If no space on the forward links to the content-reqesting peer nodes can be fond, the sorce node checks the forward links to the non-contentreqesting peer nodes. If space is fond available on a link, the content block is pt into the TCP bffer for the corresponding link. If there is still no space available even on the links to the non-content-reqesting peer nodes, the sorce node prses the final distribtion rote, and checks if there is space for one block available in all the delivery links to all the content-reqesting peer nodes. Combined with the receiving bffer length policy in Figre 5, this ensres that the bandwidth of the forward link does not get sqeezed by the traffic of the forward link, If space is fond, the content block is replicated and pt into the delivery link to each content-reqesting peer node. If there is no space on any of the distribtion rotes, the sorce node will wait for a short amont of time before it will retry to find an available rote for the content block again. 3.5 Operational analysis of Mtalcast: role of the redistribtion qee Using redistribtion qees and the above operational strategy for the peer and sorce nodes, Mtalcast handles anomalies sch In socket programming, the receiving bffer length may be obtained throgh ioctl() fnction call with parameter FIOREAD. Figre 6 The operation flow of the sorce node. as packet loss and network congestion dring content distribtion by adjsting the pload bandwidths of the nodes to achieve the maximm content distribtion throghpt by flly tilizing the pload bandwidth resorces of the sorce and peer nodes. We explain the optimality of the Mtalcast in the following. The content blocks between any two nodes are distribted throgh a redistribtion qee, which in or crrent implementation is a TCP connection with a certain size sending and receiving bffer. We notice from Section 3.4 that the Mtalcast sorce and peer nodes psh as many content blocks as possible into the TCP connections, ntil the TCP sending bffer is fll. The content blocks that are pending in the sending bffers of the TCP connections ensre that the network paths between any two peer nodes are flly tilized, even considering network anomalies sch as packet loss and network congestion. If there are no packet losses, new content blocks will be sent to the destination peer nodes

7 throgh the TCP connections. If there are packet losses or other network anomalies, TCP will try to recover from the network errors throgh retransmissions, and the content blocks that are pending in the TCP sending bffers will not be sent ot. The content blocks that are pending in the TCP receiving bffer of the forward link ensres that the pload bandwidth of the corresponding peer node is flly tilized. After the peer node pshes the last content block into the TCP sending bffer of the delivery links, it can retrieve the content block pending in the TCP receiving bffer, ths contine the activity of pshing blocks into the delivery links, and not wasting the pload bandwidth. In addition, the operational flows of Figre 4-6 ensre that the pload bandwidths of the sorce node and peer nodes are flly tilized, and the content distribtion rotes are selected in favor of the distribtion throgh content-reqesting peer nodes, then the distribtion throgh non content-reqesting peer nodes, and finally the direct distribtion from the sorce node. When we se Mtalcast to distribte content to contentreqesting peer nodes, if the pload bandwidth of the sorce node is low and the delivery links from the sorce to the peer nodes are not activated, then the content distribtion throghpt of Mtalcast will be the pload bandwidth B s of the sorce node. In this case, the content is sent ot of the sorce node at rate B s, where the peer nodes have sfficient pload bandwidth to send content to all content-reqesting peer nodes. Each content-reqesting peer node is receiving content at the rate of B s, as if the sorce node is only sending the content to it alone. If the pload bandwidth of the sorce node is high, and the delivery links from the sorce to the content-reqesting peer nodes are activated, then the content distribtion throghpt of Mtalcast will be the sm of the pload bandwidths of the sorce and peer nodes, mins a small portion of bandwidth wasted by sending content blocks to the noncontent-reqesting peers for redelivery, all divided by the nmber of content-reqesting nodes. As a reslt, Mtalcast achieves the maximm content distribtion throghpt calclated in eqation (), no matter what the network resorce (pload bandwidth) configration of the network is. Mtalcast also easily adapts to the changes in network bandwidth throgh the redistribtion qees of the TCP links. If a certain peer node slows down, the content blocks in its delivery links will move slowly, prompting the peer node to retrieve fewer content blocks from its forward link. This in trn cases the sorce node to send fewer content blocks to this now slowed down peer node, and to redirect the content blocks to other faster peer nodes. Alternatively, if a certain peer node speeds p, Mtalcast can likewise adjst by sending more content blocks to it. 3.6 Theoretical analysis of Mtalcast: maximizing content distribtion throghpt In this section we prove that Mtalcast is optimal for peer-topeer networks with constrained pload bandwidths. Mtalcast achieves the maximm possible throghpt in sch networks; no other system can do better. Let the graph (V,E) represent the network, with V being the set of nodes and E being the set of links (directed edges). Let s in V denote the sorce node and let T denote the sbset in E of content-reqesting nodes. Let the remaining nodes be non-contentreqesting nodes. Consider two types of capacities. Let e) be Figre 7 Edges have nit capacity. Broadcast capacity is two nits. Mlticast can achieve only one nit of throghpt. the capacity of each edge e in E, and let c ot (v) represent the pload bandwidth (otpt capacity) of each node v in V, sch that for each node v, the sm of the capacities of the edges leaving v is at most c ot (v). A ct between two nodes v, v 2 in V is a partition of V into two sets V, V 2 sch that v i is in V i, i=,2. The vale of the ct is the sm of the capacities e) on the edges e from V to V 2. It is well known that the maximm flow between s and any sink t in T achieves the minimm vale over all cts between s and t. Let C t be the vale of the maximm flow (the maxflow) between s and t. ote that C t = C t (c) depends on the edge capacity fnction c:e [0, ). Definition. The broadcast capacity between s and T is the minimm maxflow between s and any t in T, that is, C = min t C t. ote that like C t, C = C(c) depends on the edge capacity fnction c. Clearly, the broadcast capacity C is an pper bond on the maximm rate at which common information can be broadcast from s to all nodes in T. Unfortnately, C is not achievable in general sing mlticast roting, as the example in Figre 7 illstrates. Althogh C can always be achieved sing network coding [25], network coding reqires the intermediate nodes to code, not merely rote, their inpt packets to prodce otpt packets. If only roting is sed, the maximm throghpt C 0 from s to T via mltiple mlticast trees can be a factor of log lower than C [26]. Moreover, determining the optimal collection of mlticast trees (achieving C 0 ) is P-hard, while the tightest known bond on the gap between C 0 and the throghpt C 00 C 0 achievable in polynomial time is relatively loose [27]. On the other hand, if there are no Steiner nodes in the network (a Steiner node is a node v for which C v < C) then the broadcast capacity C can be simply achieved by greedily packing mltiple mlticast trees, as implied by Edmonds theorem [28]. Mtalcast, which is a particlarly strctred collection of mltiple mlticast trees, achieves the broadcast capacity C = C(c) for some edge capacity fnction e). Frthermore, it achieves the maximm sch broadcast capacity, as the following theorem shows. Theorem. The Mtalcast throghpt θ achieves the maximm possible broadcast capacity sbject to the node otpt capacity constraints, that is, θ = max c C(c) over all edge capacity fnctions c:e [0, ) sch that for all nodes v, the sm of e) over all edges e leaving v is at most c ot (v). Proof. We have separate proofs for networks in which B s B s + B s2 and networks in which B s B s + B s2. We prove the former with a ct separating s from V s and we prove the latter with cts separating V t from t.

8 First assme B s B s + B s2. For any edge capacity fnction c, the broadcast capacity C(c) can be at most eqal to the vale of the ct separating s from V s. Since this is at most B s c ot (s), we have max c C(c) B s. Of corse, a throghpt θ mst satisfy θ max c C(c). On the other hand, according to (), Mtalcast achieves throghpt θ = B s. Hence θ = max c C(c) = B s. ow assme B s B s + B s2. For any edge capacity fnction c, the sm of e) over all edges entering nodes in T mst be at least times the broadcast capacity C(c). Ths we have (denoting U = V T s as the set of non-content-receiving nodes): C ( c) = = t T e In ( t) e In e) e Ot c ot e) e) e In ( ) e In ( ) e In ( ) e) e) e). On the other hand, from () we have (denoting B v = c ot (v)): θ = B + + t B Bs Bt B t T t T Hence θ = = = t T t T B + c t ot B + t B + B B B + B s s B t T Bt Of corse, θ max c C(c), so θ max c C(c) = C(c*), where c* is an optimizing capacity fnction. Ths B cot = θ max x C( c) = cot c*( e). e In( ) We are done if we can show that the ineqality holds with eqality. Certainly this is tre if U is empty. To show this when U is not empty, we arge that for each in U, B Otherwise, any flow throgh to the content-receiving nodes wold c*( e). e In( ) be insfficient to se p the pload bandwidth B, whence we cold achieve a higher throghpt by re-allocating some capacity from edges between s and T to edges between s and U. Corollary. In a file download scenario, Mtalcast minimizes the maximm download time experienced by any contentreceiving peer node, and in a streaming media scenario, Mtalcast maximizes the minimm qality experienced by any contentreceiving peer node. Ths Mtalcast is ideal in sitations where a distribted grop of friends wishes to experience downloaded or streamed content at the same time with the same qality. 3.7 Mtalcast: throghpt nder download bandwidth or link bandwidth constraints The above sections assme that the only bottleneck in Mtalcast is the pload bandwidths of the peer nodes. Here we give a B. brief discssion on the Mtalcast throghpt nder link bandwidth or download bandwidth constraints. Consider a peer node i with pload bandwidth B i. Let its link bandwidth to the content-receiving peer node j be B l ij, j=0,,m-, where M is the nmber of content-receiving nodes other than itself. The link bandwidth between node i and j will not be the bottleneck as long as: l B ij B i / M. If the above ineqality is not satisfied, the pload bandwidth of node i cannot be flly tilized in the crrent Mtalcast scheme. The effective pload bandwidth of node i becomes: l B i ' = M min B. This effective pload bandwidth can be sed in eqation () to obtain the new Mtalcast ij j throghpt. When a content-receiving peer node has download bandwidth less than the throghpt given in eqation () (which is based only on the pload bandwidths), sch a node will also be a bottleneck of Mtalcast. In sch scenario, the overall Mtalcast throghpt will be the minimm download bandwidth of all the content-receiving peer nodes. This is becase all nodes have to wait for the slowest node to finish before they can resme delivery. An alternative strategy to the crrent Mtalcast implementation is to let the slow peer nodes skip certain content blocks, so that they will not slow down the receiving operation of the remaining peer nodes, which can still proceed at fll speed. In a file download scenario, the slow peer nodes may be able to receive the skipped content after all the remaining nodes have finished downloading. In a streaming media scenario, the slow peer nodes may be able to receive their content with lower qality, if layered media coding is sed. In comparison to the alternative approach, the crrent Mtalcast implementation maximizes the throghpt of common information to all content-receiving peer nodes. It maximizes the minimm qality experienced by any contentreceiving peer node in a streaming media scenario, or minimizes the maximm download time experienced by any contentreceiving peer node in a file download scenario (for example, if a distribted grop of friends wishes to experience downloaded or streamed content at the same time with the same qality). We believe in most applications, this is the preferred soltion. 4. EXPERIMETAL RESULTS A Mtalcast file distribtion soltion has been implemented. The soltion incldes a sender modle rn by the sorce node and a receiver modle rn by each of the peer nodes. To verify the performance of Mtalcast, we set p a Mtalcast content delivery network with one sorce node and for content-receiving peer nodes. Each node is attached with an pload bandwidth gag which may control its pload bandwidth. A media file of size 6MB is then broadcasted via Mtalcast from the sorce to all the peer nodes. In the first experiment, we evalate the overall effectiveness of the Mtalcast system. The pload bandwidths of the sorce and the content-receiving peer nodes are kept constant throghot the entire experimental session. We then compare the theoretical broadcast capacity verss the actal Mtalcast throghpt, which is measred by dividing the size of the distribted content by the time of distribtion. A total of for different experiments have been condcted with difference sorce and peer node bandwidths,

9 (a) (b) (c) (e) (d) (f) Figre 8 Mtalcast throghpt analysis. The horizontal axis records the time (in seconds), and the vertical axis records the bandwidth (in kbps). The figres are: a) throghpt of the sorce node s, b)-e) throghpt of the content-receiving peer nodes t -t 4, and f) the analytical throghpt (marked with a solid line) and the actal throghpt (marked with an x ). and the reslts are shown in Table 2. With jst the simple implementation of the Mtalcast sender and receiver components according to Figre 4-6, the actal Mtalcast throghpt is remarkably close to the analytical broadcast capacity of the peer-topeer network. Table 2 The throghpt of content distribtion: the analytical broadcast capacity vs. actal Mtalcast throghpt o. Upload Bandwidths (kbps) Throghpt (kbps) S t t 2 t 3 t 4 Analytical Mtalcast In the second experiment, we frther demonstrate the capability and the flexibility of the Mtalcast network in response to network change. In the experiment, the pload bandwidths of the sorce and the peer nodes may randomly adjst every second with one of three modes: 50% pward, 33% downward, or constant, all with a /3 probability. The instantaneos pload bandwidths of the sorce node and the for content-receiving peer nodes are shown in Figre 8 a)-e), where the horizontal axes denote elapsed time in seconds, and the vertical axes denote the pload bandwidths in kbps. In Figre 8f), we again compare the theoretical broadcast capacity (shown with a solid line) verss the actal Mtalcast throghpt (shown with an x ), which is measred by conting the nmber of bytes delivered every second. The horizontal axis is again the timeline, and the vertical axis is the throghpt in kbps. It is clearly demonstrated that the Mtalcast not only can achieve the analytical broadcast capacity, bt also can achieve the capacity nder complex and varying network conditions. By sing the TCP sending and receiving bffer as a redistribtion qee, Mtalcast can adapt to changing network conditions by flexibly assigning more content blocks to nodes experiencing better network conditions at the time. It achieves the broadcast capacity throghot the content distribtion session. 5. COCLUSIOS A simple yet flexible content distribtion approach called Mtalcast is developed in the paper. Mtalcast splits the to-bedistribted content into many small blocks, and assigns the distribtion of each content block to a single node, which can be a content-reqesting node, a non-content-reqesting node or the sorce node. odes with more pload bandwidth can distribte more blocks, and nodes with less pload bandwidth can distribte fewer blocks. TCP connections with their sending and receiving bffers are sed by Mtalcast to control the throghpt of the distribtion, and ensre that the pload bandwidths of the all the peer nodes and sorce node are flly tilized even with network anomalies sch as packet losses and delivery jitters. Thogh simple, Mtalcast achieves the broadcast capacity of the peer-topeer network.

10 6. REFERECES [] Internet protocol mlticast, ch 43 of Internetworking Technology Handbook, cisintwk/ito_doc/index.htm. [2] Y. Chawathe, Scattercast: an architectre for internet broadcast distribtion as an infrastrctre service. PhD thesis, University of California, Berkeley, Agst [3] J. Jannotti, D. K. Gifford, K. L. Johnson, M. F. Kaashoek, and J. W. O'Toole Jr., Overcast: reliable mlticasting with an overlay network. In Proc. of the Forth Symposim on Operating System Design and Implementation (OSDI), October [4] Y. Ha Ch, S. Rao, H. Zhang, A case for end system mlticast, In Proc. of Sigmetrics 2000, Santa Clara, CA, Jne [5] V.. Padmanabhan and K. Sripanidklchai, The Case for Cooperative etworking, In Proc. of the First International Workshop on Peer-to-Peer Systems (IPTPS), Cambridge, MA, USA, March [6] M. Castro, P. Drschel, A-M. Kermarrec, A. andi, A. Rowstron and A. Singh, "SplitStream: High-bandwidth content distribtion in a cooperative environment", In Proc. of the International Workshop on Peer-to-Peer Systems, Berkeley, CA, Febrary, [7] A. Rowstron and P. Drschel, "Pastry: scalable, distribted object location and roting for large-scale peer-to-peer systems", In Proc. of IFIP/ACM International Conference on Distribted Systems Platforms (Middleware), Heidelberg, Germany, pages , ovember, 200. [8] B. Banerjee, B. Bhattacharjee and C. Kommareddy, Scalable application layer mlticast, Proc. SIGCOMM 02, [9] M. Castro, P. Drschel, A. M. Kermarrec and A. Rowstron, SCRIBE: a large-scale and decentralized application-level mlticast infrastrctre, IEEE Jornal on Selected Areas in Commnications, pp.00-0, Vol. 20, o. 8, [0] S. Q. Zhang, B. Y. Zhao, A. D. Joseph, R. H. Katz and J. Kbiatowicz, Bayex: an architectre for scalable and falttolerant wide-area data dissemination, Proc. of OSSDAV 0, 200. [] S. Ratnasamy, M. Handley, R. Karp and S. Shenker, Application-level mlticast sing content-addressable networks, Proc. of GC 0, 200. [2] B. Zhao, J. Kbiatowicz, and A. Joseph, Tapestry: An infrastrctre for falt-resilient wide-area location and roting, U. C. Berkeley, Tech. Rep. UCB//CSD-0-4, April 200. [3] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker, A Scalable Content-Addressable etwork, Proc. of ACM SIGCOMM, Ag [4] Y. Ch, S. Rao and H. Zhang, A case for end system mlticast, Proc. of ACM SIGMETRICS 00, [5] D. Pendarakis, S. Shi, D. Verma and M. Waldvogel, ALMI: an application level mlticast infrastrctre, 3rd USEIX Symposim on Internet Technologies, 200. [6] M. Castro, M. Jones, A.-M. Kermarrec, A. Rowstron, M. Theimer, H. Wang, and A. Wolman, An evalation of scalable application-level mlticast bilt sing peer-to-peer overlay networks, Proc. of IFOCOM'03, [7] L. Cherkasova and J. Lee, FastReplica: Efficient Large File Distribtion within Content Delivery etworks, In Proc. of the 4-th USEIX Symposim on Internet Technologies and Systems, Seattle, Washington, March 26-28, [8] D. Kostic, A. Rodrigez, J. Albrecht, A. Vahdat, Bllet: High Bandwidth Data Dissemination Using an Overlay Mesh, In Proc. 9th ACM Symposim on Operating Systems Principles, October 9-22, 2003, the Sagamore, ew York. [9] J. Byers, J. Considine, M. Mitzenmacher, and S. Rost, Informed Content Delivery across Adaptive Overlay etworks, Proc. of ACM SIGCOMM, Ag [20] B. Cohen, Incentives bild robstness in BitTorrent, [2] S. Floyd, V. Jacobson, C. Li, S. McCanne, and L. Zhang, A Reliable Mlticast Framework for Light-weight Sessions and Application Level Framing, IEEE/ACM Trans. etworking, Vol. 5, o. 6, pp , Dec [22] S. Pal, K. K. Sabnani, J. C. Lin, and S. Bhattacharyya, Reliable Mlticast Transport Protocol (RMTP), IEEE J. Selected Areas in Commnications, Vol. 5, o. 3, pp , Apr [23] K. Obraczka, Mlticast Transport Protocols: A Srvey and Taxonomy, IEEE Commnications Magazine, pp , Jan [24] S. B. Wicker, V. K. Bhargava, Reed-Solomon Codes and their Applications, IEEE Press, ew York, 994. [25] R. Alswede,. Cai, S.-Y. R. Li, and R. W. Yeng, etwork information flow, IEEE Trans. Information Theory, Vol 46, o. 4, pp , Jly [26] S. Jaggi, P. Sanders, P. A. Cho, M. Effros, S. Egner, K. Jain, and L. Tolhizen, Polynomial time algorithms for network code constrction, to appear in IEEE Trans. Information Theory. [27] K. Jain, M. Mahdian, and M. R. Salavatipor, Packing Steiner trees, 4th ACM-SIAM Symp. Discrete Algorithms (SODA), [28] J. Edmonds, Edge-disjoint branchings, in Combinatorial Algorithms, R. Rstin, ed., pp. 9-96, Academic Press, Y, 973.