ISP-Supported Traffic Reduction for Application-Level Multicast

ISP-Supported Traffic Reduction for Application-Level Multicast Piotr Wydrych and Piotr Chołda Department of Telecommunications, AGH University of Science and Technology, Kraków, Poland E-mail: {piotr.wydrych,piotr.cholda}@agh.edu.pl Abstract The paper proves that it is possible to optimize application-level multicast operation from the viewpoint of traffic flows. A modification of the FreePastry/Scribe application is proposed to enable cooperation with the IETF ALTO (Application- Layer Traffic Optimization) protocol. Consequently, the overlay topology is constructed taking into account the underlying network topology. The presented results show that costly traffic types can be reduced while the increase of the delay is not harmful. Index Terms ALTO, application-level multicast, P2P-ISP cooperation, Pastry, Scribe. I. INTRODUCTION The success of file-sharing peer-to-peer (P2P) systems caused that the idea of the end-user collaboration was introduced in other fields. Among others, P2P overlays are used to create multicast distribution systems independent of multicast support in underlying networks routers. These solutions are called application-level multicast (ALM) systems. The success of peer-to-peer systems generated also a significant growth of the network operators costs. Peers forming overlay structures are not aware of the underlying network topology. Therefore, this traffic often crosses autonomous system boundaries when it is not necessary, i.e., when the same content could be reached within the same domain. This fact induces the research on topology-aware applications and cooperation between P2P users and Internet Service Providers (ISPs). These investigations [1] [3] focus on optimal peer selection in a file-sharing system. The topic of routing proximity in Distributed Hash Tables was discussed in [4] and [5]. Ratnasamy et al. [4] proposed three ways of optimizing traffic of structured peer-to-peer networks: proximity routing, proximity neighbor selection, and geographic layout. Applications using routing proximity choose the next hop according to not only the overlay metric, but also to the information concerning the physical network structure. Proximity neighbor selection is possible in networks which allow freedom of choosing peers to be inserted into routing tables (like in Pastry [6] or Kademlia [7]). The difference between these two approaches is that in the proximity routing scheme the information on the underlay topology is evaluated during the message routing process, while in proximity neighbor selection such information is taken into account during the routing table maintenance. With geographic layout, the identifiers in the overlay are assigned with regard to the underlying network topology, i.e., when two nodes are located nearby in the underlay they are given close identifiers in the overlay address space. These approaches were compared and discussed by Castro et al. [5]. Optimization of traffic generated by application-level multicast systems built on top of structured systems have not been studied intensively. Castro et al. [8] compared multicast networks built on top of two structured overlays: CAN [9] and Pastry. The results showed that assigning CAN identifiers according to the underlying topology reduces the relative delay penalty (RDP) 1 by 3% to 4%. In the tree-based CAN the selection of the routing metric was influencing the performance. For Pastry, it turned out that it is possible to reduce the link stress 2 and RDP when the messages were flooded all over the network. However, when the multicast applications were forming a tree, only proximity neighbor selection guaranteed good results. The work assumes that all nodes are provided with the exact knowledge of the underlying network. Such an assumption cannot, however, be met in existing distributed systems. Kovačević et al. [1] presented a structured treebased location-aware overlay network optimized for ALM. The system uses the Global Positioning System to determine the coordinates of the nodes and to assign the identifiers. Moreover, the capabilities of the nodes are investigated to select superpeers, that are assigned responsibility zones. However, each node of the system needs to be provided with its geographic coordinates to function properly. Picconi and Massoulie [11] showed that the optimization of the traffic generated by ALM can be successfully performed. Here, we propose to modify a P2P-based ALM application, Scribe [12], built on top of Pastry [6]. The modification aimes at another than default mode for Pastry structure construction to be able to consider the topology of the underlying IP network. The solution uses the most precise and reliable source of the information about the network topology, that is network operators routing databases. We assume that the information on network topology is provided by ALTO servers run by ISPs to optimize and communicate with the application. The ALTO protocol is being developed by IETF ALTO (Application- Layer Traffic Optimization) Working Group [13] launched in November 28 to standardize a service for better-thanrandom peer selection in overlay networks. Then, on the basis of extensive simulation experiments we prove that the traffic 1 It is the ratio of the experienced delay between the source and the receiver to the unicast delay between these two nodes. 2 It is the number the same content is sent through the same link.

1 2 3 1 213 23..113.11 198.51.1.123 131 192..2.198 21213 198.51.1.99 198.51.1.123 22 332 192..2.194 2321 23..113.238 Standard routing table maintenance process selects node 2231, since it has lower RTT 2231 23..113.168 another AS RTT=3 ms 2 21 198.51.1.174 198.51.1.123 2231 192..2.79 3 4 211 23..113.127 198.51.1.123 198.51.1.123 ALTO-enabled routing table maintenance process selects node 2212, since it is in the same AS 2212 198.51.1.87 the same AS RTT=35 ms Fig. 1. The selection of alternatives to be inserted into a routing table of node with ID 2132 by standard and modified routing table maintenance process in a simplified FreePastry network with l =1and b =2. reduction of the most costly types of traffic is observed. Additionally, the induced delay is reported to show that it is not harmful. Section II presents the proposed modification of the Free- Pastry/Scribe application. Section III presents and assesses the results obtained. Finally, Section IV concludes the paper. II. ALM TRAFFIC OPTIMIZATION WITH ALTO PROTOCOL Pastry [6] is a completely decentralized structured overlay network which uses prefix routing to forward messages. Every node and every data item is assigned its own unique l-bit identifier. Identifiers are interpreted as strings of digits to the base 2 b. Every Pastry node stores information on other nodes in three structures: a routing table, aleaf set, and a neighborhood set. Each routing table comprises l /b rows with 2 b 1 entries each. An entry in row n and column m refers to a node whose identifier shares first n digits with the identifier of a given node, but does not share the (n+1)th digit, and the (n +1)th digit is equal to m. When no node with a matching identifier is known, the routing table entry remains empty. Such a situation may particularly be common in the case of the most bottom rows (related to peers most proximate in the ID space), depending on the number of Pastry peers forming the overlay. A leaf set L contains references to L nodes numerically closest to the given node. It is divided equally into two parts. One part contains references to L /2 nodes with identifiers lower than the given node s identifier, and the other part contains references to L /2 nodes with higher identifiers. Neighborhood sets were designed to store information concerning nodes that are close to a particular node with respect to a network proximity metric (e.g. the Round Trip Time). However, in the FreePastry implementation [14], the information on proximity is maintained for every known node and is stored within the routing table. Therefore, the neighborhood sets have never been implemented. The message routing algorithm comprises two phases. First, the node, which has to make a decision to which peer it should pass a message, checks whether the message destination identifier falls within the node s leaf set. If so, the message is forwarded to the peer whose identifier is numerically closest to the destination identifier. If the destination is not within the range of the leaf set, the routing table is used to select the next hop. The message is forwarded to a peer whose identifier shares a longer common prefix with the destination than with the current node. If no matching peer is found, the message is forwarded to a peer whose identifier is numerically closest to the message destination. A. FreePastry Routing Table Optimization The standard routing table maintenance process in Free- Pastry was modified by us to benefit from external information on the proximity. The basic FreePastry system measures Round Trip Time values and uses them as the proximity assessment (Figure 1). If a peer supports ALTO, and the domain it belongs to contains an enabled ALTO server, it periodically starts the routing table optimization process. When the routing table optimization is to be started, the node requests some of its neighbors to send some of their routing table entries. The accuracy of the optimization is tuned by the number of the exchanged routing table rows, n. The algorithm is presented in Figure 2. 1) For each entry in each row i of the top n rows in the node s routing table, the peer sends a RequestRouteRow message, requesting information about row i 1. Thus, the node requests nodes which are known to it for their replacement in its routing table. 2) The FreePastry node stores, in its cache, the relationships between the obtained FreePastry node handles and the network addresses. 3) Then, it asks the ALTO 3 server to sort the network addresses. 3 Since the ALTO protocol [15] is used, there is no risk that an ISP will reveal the confidential information on its network topology.

2 Store pairs of node IDs and IP addresses 4 Replace suboptimal entries in the routing table BroadcastRouteRow 5 Trigger Scribe resubscription process ALTO-enabled node RequestRouteRow 1 Get information on other peer s IDs and IP addresses Known peers from top n rows of the nodes s routing table 3 Sort IP addresses ALTOResponse ALTOQuery ALTO server w/information about topology Fig. 2. FreePastry routing table optimization algorithm. 4) The node handles are added to the routing table in the order returned by the server, replacing the suboptimal entries. 5) The modified Scribe implementation (see Section II-B) is notified that the routing table was optimized. The modified routing protocol is designed to be fully compatible with the standard protocol and the ALTO-enabled applications can be easily deployed in an existing environment. B. Scribe Client Modifications The Scribe implementation has been modified to make the application benefit from the routing table optimization. For each multicast group, Scribe builds a tree using reverse-path forwarding [16]. Each group is totally independent of other groups, and every node can join, be root of, and publish to any number of groups. A node with an identifier numerically closest to the group identifier is the root of the multicast tree and is called a rendezvous point. In the original Free- Pastry/Scribe implementation, when a client subscribes to a topic and receives a SubscribeAckMessage, itsavesthe information about its parent in the topic tree. However, it never checks if it should resubscribe to another parent node. The modified implementation of Scribe observes all changes in the routing table and if a removal of a parent of the node in any known topic tree occurs, the application waits until the routing table optimization process finishes and then resubscribes to the topic, probably via another parent. III. SIMULATION RESULTS ASSESSMENT The simulation software was developed in Java, using ProtoPeer simulator 4 [17] and FreePastry [14] libraries, and 4 The ProtoPeer engine was preferred over the FreePastry discrete event simulator [14] for two reasons: (1) the FreePastry discrete event simulator does not allow to model and monitor the underlay in the detailed way, and (2) the ProtoPeer engine adds the ability to compare the obtained results with the results obtained during further research on other protocols. the transit-stub network model [18]. The network topology consists of four transit domains, containing five routers on average. Each transit router is connected to two stub domains on average. Each stub domain consists of five routers on average. The network consists of 22 routers connected by 334 bidirectional links. The basic features of BGP and OSPF are employed to calculate the routes. All hosts are attached to the stub routers. Two groups of simulation scenarios were prepared to check the influence of the popularity of ALTO among ISPs and end-users. The first case occurs when all ISPs provide ALTO-enabled servers and some of the clients want to use them. The selected values of the probability of creating ALTO-enabled peers were equal to: %, 25%, 5%, 75%, and 1%. The second situation takes place when all end-users run ALTO-enabled clients, but only a portion of all ISPs wants to optimize their traffic. The same five probability values were selected for the popularity of ALTO within ISPs. The next parameter varied within scenarios was the number of exchanged routing table rows, i.e., the accuracy of the optimization. In the first situation, nodes exchanged only one row, which covered statistically 1 ( 1 2 3 ) 1 =87.5% of the identifier space. In the second case, nodes exchanged two rows, which held statistically 1 ( 1 2 3 ) 2 =98.4% of the identifiers. The third set of the simulations was configured to exchange four rows, which influenced statistically 1 ( 1 2 3 ) 4 =99.98% of the identifiers. Each simulation ran for six hours. The first Scribe peer started at the beginning of the simulation run. Peers started one after another with delays that were randomized from the range (, 5 s). During the simulation experiments, every ALTO-enabled peer repeated its ALTO query process every 2 minutes. After 5 seconds, a randomly selected peer stopped. The peer stop interval was randomized from the same range. The multicast traffic source was never stopped.

Link stress 7 6 5 4 3 2 1 % 25% 5% 75% 1% ALTO popularity Link stress 1 9 8 7 6 5 4 3 2 1 Without ALTO server With ALTO server % 25% 5% 75% 1% ALTO server popularity Fig. 3. Link stress: global traffic. Each point represents one link. Fig. 4. Link stress: incoming transit traffic. ALTO client popularity = 1%. This resulted in the average number of running peers equal to 1995, after a warm-up period equal, on average, to 1 hour 23 minutes. The presented values were averaged separately for each link. Moreover, for each of the mean values, the 95% confidence intervals were calculated, but are not present in the figures as they are very small and could hinder the legibility. A more detailed information on the results obtained during the experiments can be accessed on the website [19]. A. Multicast Traffic The terms of the transit, peering and local traffic are used as defined in the ALTO Problem Statement [2]. 1) Global Traffic: As shown in Figure 3, introducing ALTO makes a significant impact on the volume of the traffic carried by the links in the transit domains and between them. When all stub domain operators and all clients use ALTO, the overall traffic flowing within transit domains is reduced by 38% and the overall traffic flowing between transit domains is reduced by 71%. 2) Incoming Transit Traffic: The most important benefit the stub network operators may have from using ALTO is that they can reduce their incoming transit traffic, which is very expensive. Bringing ALTO into general use reduces the traffic flowing from the transit domains to the stub ones by 22% on average. What is more important, in the heterogeneous environment, when some of stub networks use ALTO and others do not, the latter do not suffer from the fact of this diversity, what can be seen in Figure 4. The volume of the traffic flowing to the domains without an ALTO server remains unchanged. In all cases when ALTO-enabled and ALTO-disabled domains coexist in the same environment, enabling the ALTO server in a domain leads the incoming transit traffic to diminish by 21% on average. 3) Outgoing Transit Traffic: The structure of outgoing transit traffic flow was influenced significantly by the ALTO popularity. In all cases, the domain containing the multicast tree root suffers from the high outgoing transit traffic. Using ALTO causes a great reduction of the volume of this unfavorable traffic, up to 8%, what was shown in Figure 5. Increasing popularity of the ALTO service brings on a growth of transit Link stress 16 14 12 1 8 6 4 2 % 25% 5% 75% 1% ALTO client popularity Fig. 5. Link stress: outgoing transit traffic. ALTO server popularity = 1%. traffic outgoing from all other stub domains to the transit ones. Nevertheless, the growth is dispersed among many different stub-to-transit links. Since Scribe uses reverse-path forwarding and clients download multicast data from their parents in the tree, there is no significant difference between volume of traffic originating from ALTO-enabled and from ALTO-disabled stub domains. 4) Peering Traffic: Optimizing the overlay traffic causes a substantial growth of the peering traffic, i.e., the traffic being exchanged directly between stub domains. It is important to remember that in most cases this traffic is much cheaper than the transit one. The increase in the average peering traffic depends linearly on the ALTO client probability, and the peering traffic volume in the case of 1% ALTO probability is about 15 times greater than in the case of no ALTO. Figure 6 shows that the growth of the incoming peering traffic significantly depends on whether the domain contains an ALTO server or not. The outgoing peering traffic volume originating from the ALTO-enabled and the ALTO-disabled domains remains at the same level. 5) Local Traffic: It is observed that even a small fraction of ALTO-enabled peers increases the non-problematic traffic which stays within one stub domain. Surprisingly, the average level of the increase of the local traffic does not depend on the fact whether the stub domain contains an ALTO server or not.

Link stress 2 15 1 5 Without ALTO server With ALTO server Average link load [Mb/s] 14 12 1 8 6 4 2 Global Transit Peering Local % 25% 5% 75% 1% ALTO server popularity Without ALTO 1 2 4 Rows exchanged with ALTO popularity=1% Fig. 6. Link stress: incoming peering traffic. ALTO client popularity = 1%. Fig. 7. Estimation of different types of traffic. B. Maintenance Traffic To assess the amount of the overhead generated by the optimization process, the maintenance messages have been also logged and analyzed. Enabling ALTO in all domains and on all peers causes a fivefold increase of the amount of the maintenance traffic flowing within stub domains. The impact of the popularity of ALTO within ISPs on the traffic flowing between ISPs is not significant (growth by 1 8%). 1) ALTO Queries and Responses: The ALTOQuery and ALTOResponse messages are exchanged between peers and ALTO servers. Peers contact only ALTO servers in their domains. Thus, the ALTO query and response messages are observed only on the intra-domain links inside the stub domains. The intensity of the messages is growing linearly with the popularity of the ALTO clients. What is obvious, the query and response messages are not observed within the domains that are not running ALTO servers. Moreover, the number of such messages passing the links inside an ALTO-enabled domain does not depend on the popularity of ALTO among all other network operators. 2) Route Row Requests and Responses: The RequestRouteRow and BroadcastRouteRow messages are standard FreePastry messages used by the peers to exchange information on other peers addresses. They are used by the FreePastry ALTO implementation. The intensity of the RequestRouteRow messages depends linearly on the popularity of ALTO. Since ALTOenabled peers send requests to both ALTO-enabled and ALTOdisabled peers, without distinguishing between them, the RequestRouteRow messages are observed also within the ALTO-disabled domains. The BroadcastRouteRow messages are used also by the standard FreePastry routing protocol. Therefore, such messages were observed even before introducing ALTO. Similarly to the case of RequestRouteRow messages, the BroadcastRouteRow message intensity was growing with the increase of the ALTO popularity. 3) Scribe Messages: The SubscribeAck messages are sent only when a peer joins a tree. Therefore, without traffic optimization processes, peers join the topic only on their start and such acknowledgments are sent rarely. Growth of the popularity of ALTO increases the frequency of resubscribe events on a scale of the whole network. Since the unsubscribe and drop messages are used during the resubscribing process, the linear growth of the intensity of both message types is observed. C. Total Traffic Estimation The maintenance traffic growth does not overcome the multicast traffic decrease. It was assumed that the multicast stream bit rate is 128 kb/s and that the stream was divided into 1-byte frames. The multicast and maintenance flows were summed up for each type of traffic separately. Figure 7 presents the impact of introducing ALTO on different types of traffic. Moreover, it is shown how the number of the rows exchanged during the FreePastry routing optimization process influences the data rates. For better readability, only two extreme cases were presented: when there are no ALTO servers, and when all clients and all domains optimize their traffic. In other cases, the volume of the global, transit, and peering traffic depends linearly on the ALTO popularity. The increase of the local traffic is significant even for the low (25%) ALTO popularity. It turns out that the number of exchanged rows does not influence the reduction of the global traffic. When considering the three other traffic types, the best results are observed when the number of exchanged rows is set to two. It is not advised to choose too large values of the accuracy of the optimization, because the maintenance traffic growth may overcome the multicast traffic decrease. D. Relative Delay Penalty The quality of experience in ALM services is inextricably linked with the notion of relative delay penalty (RDP). Since the metric used by the ALTO servers to rank lists was designed to optimize network operators costs, and not to optimize RDP, it was predictable that RDP would grow with the increase of the ALTO popularity. However, while the increasing popularity of ALTO causes an increase of RDP in the domains without ALTO servers, RDP observed by the peers in the ALTO-enabled domains does not depend on the popularity of ALTO within all other ISPs. The comparison is

Cumulative proportion of messages 1..8.6.4 %.2 25% 5% 75%. 1 2 3 4 5 6 7 8 9 1 Relative delay penalty, RDP Fig. 8. Relative delay penalty in ALTO-disabled domains. Different curves are related to various ALTO popularity levels across the whole network. Cumulative proportion of messages 1..8.6.4 25%.2 5% 75% 1%. 1 2 3 4 5 6 7 8 9 1 Relative delay penalty, RDP Fig. 9. Relative delay penalty in ALTO-enabled domains. Different curves are related to various ALTO popularity levels across the whole network. The plot for 75% popularity overlaps with the one for 1% popularity. presented in Figures 8 and 9 showing our simulation results. A peer observes RDP equal to one when it downloads directly from the multicast content source. However, the root node may be unable to handle 8% of all the group (channel) members (see Figure 8, results for % popularity). Without ALTO, subscription requests generated by a part of users may be rejected. Using ALTO and selecting a nearby peer instead of the source as a parent significantly increases the probability that a peer will join the group and receive the requested stream. IV. CONCLUSIONS The results indicate that overlay multicast applications can be easily modified to obtain information from the ALTO servers to significantly reduce ISPs costs. The volume of the most expensive global and transit traffic can be significantly reduced, while the amount of traffic carried by peering and local links increases. The growth of those cheap types of traffic is lower than the reduction of the expensive ones. Concluding, setting up ALTO servers and implementing ALTO clients in ALM applications may lead to the significant reduction of the network operators costs. ACKNOWLEDGMENTS The authors would like to thank Andrzej Jajszczyk and Rafał Stankiewicz for their helpful comments. This work was supported in part by the Polish Ministry of Science and Higher Education under Grant N N517 555239. REFERENCES [1] V. Aggarwal, A. Feldmann, and C. Scheideler, Can ISPS and P2P users cooperate for improved performance? SIGCOMM Comput. Commun. Rev., vol. 37, pp. 29 4, Jul. 27. [2] D. R. Choffnes and F. E. Bustamante, Taming the Torrent: A Practical Approach to Reducing Cross-ISP Traffic in Peer-to-Peer Systems, SIGCOMM Comput. Commun. Rev., vol. 38, pp. 363 374, Aug. 28. [3] H. Xie, Y. R. Yang, A. Krishnamurthy, Y. G. Liu, and A. Silberschatz, P4P: Provider portal for applications, SIGCOMM Comput. Commun. Rev., vol. 38, pp. 351 362, Aug. 28. [4] S. Ratnasamy, I. Stoica, and S. Shenker, Routing Algorithms for DHTs: Some Open Questions, in Peer-to-Peer Systems, ser. LNCS, P. Druschel, F. Kaashoek, and A. Rowstron, Eds. Springer, 22, vol. 2429, pp. 45 52. [5] M. Castro, P. Druschel, Y. C. Hu, and A. Rowstron, Topology- Aware Routing in Structured Peer-to-Peer Overlay Networks, Microsoft Research, Technical Report MSR-TR-22-82, 22. [6] A. Rowstron and P. Druschel, Pastry: Scalable, Decentralized Object Location and Routing for Large-Scale Peer-to-Peer Systems, in Proc. IFIP/ACM International Conference on Distributed Systems Platforms Middleware 21, Heidelberg, Germany, Nov. 21, pp. 329 35. [7] P. Maymounkov and D. Mazières, Kademlia: A Peer-to-Peer Information System Based on the XOR Metric, in Proc. International Workshop on Peer-to-Peer Systems IPTPS 2, Cambridge, MA, USA, Mar. 22. [8] M. Castro, M. B. Jones, A. M. Kermarrec, A. Rowstron, M. Theimer, H. Wang, and A. Wolman, An Evaluation of Scalable Application-Level Multicast Built Using Peer-to-Peer Overlays, in Proc. 22nd Annual Joint Conference of the IEEE Computer and Communications Societies INFOCOM 23, vol. 2, San Francisco, CA, USA, Mar. Apr. 23, pp. 151 152. [9] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker, A Scalable Content-Addressable Network, SIGCOMM Comput. Commun. Rev., vol. 31, pp. 161 172, Aug. 21. [1] A. Kovačević, O. Heckmann, N. Liebau, and R. Steinmetz, Location Awareness Improving Distributed Multimedia Communication, Proceedings of the IEEE, vol. 96, no. 1, pp. 131 142, Jan. 28. [11] F. Picconi and L. Massoulie, ISP Friend or Foe? Making P2P Live Streaming ISP-Aware, in Proc. 29th IEEE Int. Conf. Distributed Computing Systems ICDCS 9, 29, pp. 413 422. [12] M. Castro, P. Druschel, A. M. Kermarrec, and A. I. T. Rowstron, Scribe: a Large-Scale and Decentralized Application-Level Multicast Infrastructure, IEEE Journal on Selected Areas in Communications, vol. 2, no. 8, pp. 1489 1499, 22. [13] IETF, Alto Status Pages. [Online]. Available: http://tools.ietf.org/wg/ alto/ [14] FreePastry. [Online]. Available: http://freepastry.org/freepastry/ [15] R. Alimi, R. Penno, and Y. Yang, ALTO Protocol, Internet Engineering Task Force, Internet-Draft draft-ietf-alto-protocol-6, Oct. 21, work in progress. [16] Y. K. Dalal and R. M. Metcalfe, Reverse Path Forwarding of Broadcast Packets, Communications of the ACM, vol. 21, no. 12, pp. 14 148, 1978. [17] ProtoPeer The P2P Prototyping Toolkit. [Online]. Available: http://protopeer.epfl.ch/ [18] E. Zegura, K. Calvert, and S. Bhattacharjee, How to Model an Internetwork, in Proc. 15th Annual Joint Conference of the IEEE Computer and Communications Societies INFOCOM 96, vol. 2, San Francisco, CA, Mar. 1996, pp. 594 62. [19] P. Wydrych, ALTO+Scribe. [Online]. Available: http://www.kt.agh. edu.pl/~wydrych/research/altoscribe/ [2] J. Seedorf and E. Burger, Application-Layer Traffic Optimization (ALTO) Problem Statement, RFC 5693 (Informational), Internet Engineering Task Force, Oct. 29.