Exploiting Local Popularity To Prune Routing Indices In Peer-to-Peer System

Exploiting Local Popularity To Prune Routing Indices In Peer-to-Peer System Stephane Bressan Department of Computer Science National University of Singapore steph@comp.nus.edu.sg Achmad Nizar Hidayanto Faculty of Computer Science University of Indonesia {nizar, zhasibua}@cs.ui.ac.id Zainal A. Hasibuan Abstract Routing in unstructured peer-to-peer system relies either on broadcasting (also called flooding) or on Routing Indices. An approach using Routing Indices is only scalable if the Routing Indices are of manageable size. In this paper, we present a strategy to prune Routing Indices based on the popularity of resources. The Routing Indices maintain routing information for queries to the most popular resources only. Queries to other resources are routed randomly. The popularity of resources is learned at each node of the network for each Routing Index thanks to a replacement strategy. We comparatively evaluate the performance of the local popularity method with the one of a global popularity method for the pruning of Routing Indices in both static and dynamic environments. We compare the effectiveness of two standard replacement strategies: Least Frequently Used (LFU) and Least Recently Used (LRU). Our results confirm the efficiency and effectiveness of the pruning of Routing Indices based on local popularity of resources in unstructured peer-to-peer networks. I. INTRODUCTION Peer-to-peer technology can be divided into two main classes according to the level of organization of the nodes and resources it assumes: namely structured and unstructured peerto-peer architectures. In the structured architectures such as [1], [2], [3], nodes are connected to each other and resources or the records are moved according to some mapping techniques (usually hashing). In unstructured architectures such as [4], [5], nodes are connected to each other as they join the network and resources are not being moved but hosted on site. Hence, searching for a resource requires to either broadcast the request or use routing information. The design of Routing Index then becomes a critical aspect of an efficient, effective and scalable system. Our contribution in this paper is two-fold: 1) a general approach to prune Routing Indices by learning the popularity of resources in the local context, 2) we comparatively evaluate the performance of the local popularity method with the one of a global popularity method for the pruning of Routing Indices in both static and dynamic environments. The pruning based on global popularity is a method we have presented and studied in [6]. In the next section we present a brief overview of the main related work on searching and routing strategies in unstructured peer-to-peer systems. As in previous works on routing in unstructured peer-to-peer systems (see [6] and [7]), we are using a reinforcement learning approach to learn the routing information called Q-routing. We briefly recall the principles underlying reinforcement learning in section III. We present the Routing Indices, routing strategy and proposed strategy to prune Routing Indices in section IV. In section V we evaluate and analyze the performance of the approach we propose under static and dynamic environments. In the last section we present our conclusions. II. RELATED WORKS Let us first recall the outline of the respective architectures of the two main operational unstructured peer-to-peer systems: Gnutella [4] and Freenet [5]. Gnutella is a decentralized filesharing system whose participants form a virtual network communicating from peer to peer via the Gnutella protocol, which consists of 5 basic message types: ping, pong, query, queryhit and push. These messages are routed to their destination using constrained broadcast by decrementing the message s time-tolive field. Freenet, an anonymous peer-to-peer storage system, is decentralized, symmetric and automatically adapts when nodes leaving and joining. Freenet does not assign responsibility for documents to specific servers; instead, its lookups take the form of searches for cached copies. This allows Freenet to provide a degree of anonymity, but prevents it from guaranteeing retrieval of existing documents or from providing low bounds on retrieval costs. The authors of [8] have introduced a more interesting concept than constrained broadcasting for the routing of requests in unstructured peer-to-peer systems, namely Routing Indices (RI). RI are tables located and used at individual nodes of the network to forward queries to designated (by the content of the table) neighbors that are likely to have the answers needed. When a query arrived at a node, the node will use the RI to select neighbor nodes and forward the query to it. The authors of [8] proposed 3 different types of RI: Compound

2 RI, Hop-count RI, and Exponential RI. Compound RI provides the number of resources that can be obtained when a query was forwarded to a selected neighbor. While Compound RI provides only information about the number of resources, it provides no distance information. Hop-count RI added the distance information to the RI. Exponential RI also provides both kind of information but present them in a different way than the one of Hop-count RI. The RI are constructed by propagating local information to neighboring nodes at a fixed interval or when the changes in the local indices have reached a certain threshold. In the proposed scheme, resources in the network are classified into different classes and indexing is done with these classes. While this reduces the size of the index, it prevents finer granularity. We have shown in [6] and [7] that Routing Indices can be made adaptive thanks to a reinforcement learning approach to their construction and maintenance. III. BACKGROUND A. Peer-to-Peer Search and Routing Indices Generally, searching mechanism in peer-to-peer systems can be divided into two main components: resource locating and routing. In resource locating, given a resource id, the challenge is to locate the resource in a minimal time to yield better performance and response time. In routing, the challenge is to efficiently route a resource request to the next node in order to achieve minimal time. This is usually done with the help of Routing Indices. Routing Indices, which are maintained at each node, store the cost of routing requests for resources via each of its neighbors. In our early work [6], we have developed an adaptive routing mechanism that relies on the Q-Routing algorithm for learning the routing information. Each node in the network maintains a Routing Index called the Q-table that, for each neighbor node, stores the estimated time that a query may take to reach a node that hosts the requested resource. We have shown that the algorithm is effective and leads to converging and efficient routing time. In [6], we experimented with various techniques for the pruning of the Routing Indices. We showed that the best criteria is the popularity of resources (as opposed for instance to the distance of their location to the local node). Yet, as noted at the time, we assumed that the distribution of the popularity of resources was known and global. This is the hypothesis that we challenge in this paper. In [7],we have extended the adaptive routing approach to keyword search. Reinforcement learning estimated both routing time and relevance. Routing is then a compromise between efficiency and effectiveness. We showed how users can tune this compromise, as well as we propose a device that automatically tune the compromise in function of the network resources. In this paper, we focus on pruning strategies for Routing Indices. These pruning strategies are very important to decrease the size of the Routing Indices at each node. We define the routing and searching problem in peer-to-peer networks as follows. We consider a set of nodes and their Routing Indices, P = (p 1, ri 1 ),..., (p n, ri n ), and a set of resources, R = r 1,..., r m, located on the nodes and identified by the set of keys K = k 1,..., k m. Given a request of the form locate(k i ), i.e. a request to locate resource r i, a node receiving this request must make a local decision as to which of its neighbors to forward the request based on information from its Routing Index. B. Reinforcement Learning A reinforcement learning agent [9] learns from its interactions with its environment. In a given state, the reinforcement learning agent selects the next action according to a model which constantly constructs and updates based on the reward that it receives when trying the different actions. A routing agent could, for instance, favor an action that yields the minimum estimated time to destination. Notice that, in order to continuously learn about its environment, the agent must occasionally depart from its base strategy and try random actions. Reinforcement learning has been successfully applied to many domains, among which network routing. There are numerous forms and variants of reinforcement learning that have been proposed for various network routing problems: Q-Routing [10], Dual Q-Routing [11], Confidence-based Q-Routing and Dual Confidence-based Q-Routing [12], etc. Experiment in [12] showed that confidence-based yields the best performance among all of them but requires large storage. For the sake of simplicity and scalability we chose Q-Routing. In Q-Routing, a node that receives a request for a resource and hosts the resource then provides the resource to the original requester otherwise forwards the request according to the reinforcement learning strategy. It adopts forward propagation strategy to estimate status of the network by maintaining a table called Q-table that contains Q-value Q(n, r) which quantifies the expected reinforcement when a request for a resource r is sent to n. Such algorithms can outperform shortest-paths algorithms such as Bellman-Ford for they are able to estimate routing time taking into account the traffic. IV. DESIGN At each node of the peer-to-peer network, we maintain a Routing Index. The values it contains are estimates of the routing time of a request to the corresponding resource. We use reinforcement learning, namely Q-routing, for the maintenance of the values stored in the Routing Indices and routing of the requests. Storage capacity at each node is limited. A Routing Index cannot store information about all possible requests. This does not matter since some requests are so infrequent (the resources are not popular) that a longer service time would not impact the overall performance of the system. Thus we can view Routing Indices as caches. We may only want to store important information that will be accessed frequently due to limited size of the cache compare to the number of information that has to be stored. Fortunately, we can use a number of page replacement algorithms to control the information that

3 should be moved in and out of the cache. Thus, we borrow the mechanism to learn the popularity of the resources in the local context. In the next subsections, we will recall our routing strategy design and our proposed pruning strategy that will be incorporated into our routing strategy. A. Routing Strategy We employ Q-Routing [10] as our routing technique. Q- Routing is an adaptive routing strategy that uses Q-learning, a reinforcement learning algorithm. Q-Routing has been studied in various network routing literatures and the performance studies showed that it is able to adapt to the network traffic. In Q-learning [9], the states and the possible actions in a given state are discrete and finite in number. It uses a set of values termed Q-values when learning. Each Q-value is denoted by Q(s, a) where s is a given state and a is the action taken. In general, Q-value represents the expected reinforcement when the action a is being taken in a state s. Given the current state of the network, Q-Routing algorithm learns and finds an optimal routing policy from the Q-values distributed over all nodes in the network. Each node p in the network has its own view of the state of the network through a table that stores all the Q-value, Q p (p, r), where r R, a set of resource objects in the peer-to-peer network and p N(p) the set of all neighbors of node p. Q p (p, r) is defined as the best estimated time that a query takes to reach the node that hosts the resource r from node p through its neighbor p. The Q-value, Q p (p, r) can be represented by the following mathematical equation: Q p (p, r) = q p + δ + Q p (p, r) (1) From the equation we can derive that the minimum time needed to locate a resource r in the peer-to-peer network from the node p through its neighbor p, is affected by 3 components: 1) The time the query spends in node p, q p. 2) The transmission delay between p and p. 3) The best estimation time for Q p (p, r). Once we are clear what the Q-values stored at each node stand for, it is easy to see how we can use them for making locally optimal routing decisions. When a node p receives a query q(s, r) looking for resource r, node p will lookup its Q- value table Q p (, r) to select the neighboring node p with the minimum Q p (p, r) value. With this mechanism being used in every node in the network, the query will be answered in a minimal time. B. Updates of Q-value The performance of the routing strategy depends on all the Q-values in the path taken by the query. The closer these Q-values to the actual time needed, the closer the routing strategy reflects the actual optimal time. Therefore, the process of updating the Q-value in a node is very important. Resources Fig. 1. r1 r2 r3 rn Neighbour Q-value p1 p2... pn 12 6... 8 10 12... 7 4 10... 9 3 4 Q-value table for node p Recall that when node p receives a query q(s, r), it will find a node in the neighbor list such that Q p fulfills the following condition: Q p (p, r) = min Q p(p, r) (2) p N(p) Once p is chosen, p will then forward the query q(s, r) to p. Upon receiving the query, p will first search if it has the requested resource r. If the requested r is not in the shared list of p, it will have to forward the query to its neighbor. For both cases, p will have to send the Q-value back to p. In the first case, since the query can be answered by p, no further routing of the query q(s, r) is needed and the Q-value returned to p is 0. While in the second case, the Q-value, Q p (p, r), returned will be according to equation 2. The Q-value returned by p is essentially its estimation of how long a query will be answered after p has processed the query. The estimation returned is excluded the queuing time at p and the transmission delay, δ, for the query to travel from p to p. Based on the estimated value, node p will calculate the new Q-value for Q p (p, r) with the following equation: Q p (p, r) new = Q p (p, r) old + l(q p (p, r) new + q p + 9 δ Q p (p, r) old ) (3) where l is the learning rate with 0 l 1. When l is set to 1, the equation will become: Q p (p, r) = Q p (p, r) + q p + δ (4) However, since the new Q-value sent back by p is the propagation of estimated values, the Q-value is not necessary accurate. Therefore, in order to reflect the possible error caused by estimation in the learning process, the learning rate should always be set to a value less than 1 {eg. 0.7 has been a suggested learning rate on previous network routing studies}. C. Pruning of Routing Indices Figure 1 depicts the structure of a Routing Index that store Q-values. Each resource r is a record row in the Routing Index. For each resource, there is corresponding Q-value for each neighbor. Since the number of resources in the network determine the number of rows, therefore, the size of the Routing Index is proportional to the resources shared in the network and it can be potentially big. In this work, we study the mechanisms to maintain the size of the Q-value table by proposing the following methods: Stores only n entries that are most recently requested Stores only n entries that are most frequently requested The proposed methods are borrowing the mechanism in the page replacement algorithms, that will be explained in the following subsections.

4 1) Global Popularity-Method: The pruning method is according to popularity of resources. This method indexes resources that are most frequently requested in the network. In this method, we assume that we have already known about the popularity of the resources all the time. In the experiment, we do this by generating a popularity list using zipfian distribution. Those resources that are not listed in the popularity list will be ignored. 2) Most Recently Used-Method: This method is based on Least Recently Used (LRU) algorithm. System tries to learn the local popularity of the resources by examining queries that come to the node. Each Routing Index then only stores n resources and their corresponding Q-values which are most recently requested. When a node receives a request on resource r, its Routing Index will be updated by replacing a resource that least recently requested with r and its corresponding Q- values. 3) Most Frequently Used-Method: This method is based on Least Frequently Used (LFU) algorithm. System tries to learn the local popularity of the resources by examining queries that come to the node. Each Routing Index then only stores n resources and their corresponding Q-values which are most frequently requested. When a node receives a request on resource r, its Routing Index will be updated by replacing a resource that least frequently requested with r and its corresponding Q-values. We add a column in the Routing Index to store frequency information. The value of the frequency column will be set to 1 when a resource is put in the first time in the Routing Index or replacing the least frequently used resource and will be incremented with 1 while the resource has resided already in the Routing Index. V. PERFORMANCE ANALYSIS We simulate the peer-to-peer network using our simulator to study the performance of our proposed pruning strategies. The network is a randomly re-wired ring lattice. A ring lattice is a graph with n vertices, each of which is connected to its nearest k neighbors. In our experiments, we use a k value of 4. The rewiring is done by randomly choosing m vertices to connect with each other. In our experiments, we use an m value of 10% of n. This relatively small number compared to n is meant to reflect the small world effect [13]. We randomly assign the d resources to the n nodes according to a uniform distribution. The workload is determined by a Zipfian distribution corresponding to the popularity of the resources. Requests for a resource are generated according to this distribution. Requests originate from a random node chosen uniformly. Figures 2 1 to 5 show the experiment results on static peerto-peer environment without node joins and leaves. We first compare the performance of the global popularity-method, the local popularity-method with LRU and LFU algorithm, as well as for reference, the performance of a method randomly preselecting the resources to be kept in every Routing Index. The workload of the resources is determined by a Zipfian with a 1 The graphs should be viewed in color Fig. 2. Query result on 50% pruning with Zipf θ 0.5 Fig. 3. Query result using LFU with Zipf θ 0.9 Fig. 4. Query result using LFU with Zipf θ 0.5 Fig. 5. Query result using LFU with Zipf θ 0.1 θ coefficient of value 0.5 and prune the Routing Index until 50% of its normal size. The experiment result shows that the LRU- and randommethods do not scale up as they can not route efficiently with 50% and less of the resources in the Routing Indices. This phenomena is not only due to the possible irrelevance of the entries in the Routing Indices but also to their high instability and therefore to the inaccuracy of the content of the Routing Indices under these approaches. Among them, LFU algorithm is the best strategy to approach the performance of the global popularity strategy. Therefore, in the next experiment we only present results for local popularity pruning method using LFU

5 Fig. 6. Dynamic leaving & joining every 100 unit of time with Zipf θ 0.5 Fig. 7. Dynamic leaving & joining every 100 unit of time on 40% pruning Fig. 8. Dynamic leaving & joining on 40% pruning with Zipf θ 0.5 algorithm. Figure 3 to 5 show the results of θ equals to 0.9, 0.5, and 0.1 respectively on various pruning factor using LFU algorithm. For each figure, we can see that different pruning factor indeed produces gradual effect on routing time. The experiments also show that different value of θ resulting different pruning factor that makes the routing time still converge. When we set θ equals 0.9, the routing time cannot converge when we prune the Routing Index up to 50%. But when we set θ equals 0.1, the routing time still achieve convergence even we prune the Routing Indices up to 80%. This result confirms that the routing algorithm we employed can be scaled. Figures 6 to 8 show the experiment results on dynamic peer-to-peer environment. To create the dynamic environment, we simulate a node joining or leaving in every t unit of time. First we observe the performance of routing time on different pruning factors with θ equals 0.5. The experiment results show that we still get good result although we are working on dynamic environment. We still can achieve convergence although we prune the Routing Indices until 40%. It can be observed that in some points the routing time is increase, that may be caused by a node leaving. But soon the system tries to update its Routing Indices, so that it can achieve stable state again that brings the routing time back to the minimum time. In the second experiment, we observe again the effect value of θ in dynamic environment. We examine the result by choosing pruning factor 40% and simulating nodes joining or leaving for every 100 unit of time. The result can be seen in figure 7. The figure clearly tells us that we still have gradual effect of the routing time although we are working on dynamic environment. The smaller the value of θ the better the routing time. Figure 8 shows our last experiment to exercise the impact of frequency of node joining and leaving to the routing time. The more node joining or leaving the more dynamic the environment. We exercise the impact by simulating a node joining or leaving every n unit of time, by setting the value of n: 15, 25, 100, 200 and 300. The figure shows that the more dynamic the environment the more difficult the system to converge its routing time. We can observe that the phenomena is caused by the instability of the Routing Indices that may lead to the non optimum path or the resources belonging to the leaving node no longer exist. Therefore, the node needs longer time to search the resource. VI. CONCLUSION AND FUTURE WORKS We have presented a pruning strategy for unstructured peerto-peer systems. The strategy learns the local popularity of the resources to prune the Routing Index in a node. We have shown by empirical analysis that the proposed strategy works well in both static and dynamic environment. This work proves that the viability (effectiveness, efficiency, and scalability) of Routing Indices. We have shown that the necessary adaptiveness can be conferred a reinforcement learning approach. REFERENCES [1] I. Stoica, R. Morris, D. Karger, F. Kaashoek, and H. Balakrishnan. Chord: A scalable Peer-To-Peer lookup service for internet applications. In Proceedings of the 2001 ACM SIGCOMM Conference, 2001. [2] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. A scalable content addressable network. In Proceedings of the 2001 ACM SIGCOMM Conference, 2001. [3] P. Druschel and A. Rowstron. Past: A large-scale, persistent peer-to-peer storage utility. In Proceedings of the Eighth Workshop on Hot Topics in Operating Systems (HotOS-VIII), May 2001. [4] Gnutella home page. In http://gnutella.wego.com. [5] I. Clarke, O. Sandberg, B. Wiley, and T. W. Hong. Freenet: A distributed anonymous information storage and retrieval system. In Lecture Notes in Computer Science, 2001. [6] C. Y. Liau, A. N. Hidayanto, and S. Bressan. Adaptive peer-to-peer routing with proximity. In Proceedings of DEXA Conference, 2003. [7] S. Bressan, A. N. Hidayanto, C. Y. Liau, and Z. A. Hasibuan. Adaptive double routing indices: Combining effectiveness and efficiency in p2p systems. In Proceedings of DEXA Conference, 2004. [8] J. Kubiatowicz et al. Routing indices for peer-to-peer systems. In Proceedings International Conference on Distributed Computing Systems, July 2002. [9] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT Press, 1998. [10] M. Littman and J. Boyan. A distributed reinforcement learning scheme for network routing. In Proceedings of the International Workshop on Applications of Neural Networks to Telecommunications, 1993. [11] S. Kumar and R. Mikkulainen. Dual reinforcement q-routing: An online adaptive routing algorithm. In Proceedings of the Artificial Neural Networks in Engineering Conference, 1998. [12] S. Kumar and R. Miikkulainen. Confidence based dual reinforcement q-routing: An adaptive online network routing algorithm. [13] A. Oram, editor. Peer-to-Peer : Harnessing the Power of Disruptive Technologies. O Reilly & Associates, 2001.