Efficient Cache Structures of IP Routers to Provide Policy Based Services

Transcription

1 Efficient Cache Structures of IP Routers to Provide Policy Based Services Shingo Ata Masayuki Murata Hideo Miyahara Abstract: The policy service based on the traffic characteristics is becoming important for the Internet shared by various applications. For this purpose, the router is required to forward packets according to the requirements of the traffic flow that the packet belongs to. However, the packet (and flow) classification easily becomes a bottleneck in the high speed router because the router is necessary to search multiple fields within the packet. In this paper, we propose a new packet classification algorithm capable of following the packet forwarding rate of the high speed routers with the commercially available CPU, RAM, and cache memories. Through our experiments, we show the effects of parameters of the cache memory on the processing rate of packet classification. 1 Introduction A rapid growth of the Internet and proliferation of new multimedia applications lead to demands of high speed networks. In the today s Internet shared by the various applications, it is necessary to realize a policy service based on the traffic characteristics. For this purpose, the router is required to forward the packets according to the requirements of the traffic flow that the packet belongs to. To provide the policy service at the router, the capability of packet classifications is necessary for the decision about an appropriate action against the flow to guarantee the user s QoS (Quality of Service) and/or the Internet Service Provider s Service Level Agreement (SLAs). However, the packet (and flow) classification would easily become a bottleneck in the high speed router because the router is necessary to search multiple fields within the packet for the flow classification. There are some studies speeding up the address lookups of the router (see, e.g., [1] and references therein). However, it is difficult to apply those proposals directly to the flow classification, because the size of a lookup table increases exponentially with the number and length of the packet field. From this reason, it is necessary to investigate the new algorithm suitable for the flow classification. A study on the flow classification can be found in [2]. However, the proposal in [2] uses a bit level parallelism to look up multiple fields simultaneously, and the customized hardware (e.g., FPGA or ASIC) is necessary to obtain the expected performance, leading to increase the cost of routers. If we can use a general purpose CPU, on the other hand, the total cost of the flow classification capable routers can keep low since the cost of commercially available CPU becomes more and more reasonable. When the flow classification mechanism is realized by the general purpose CPU, the latency of memory accesses is most dominant in determining the performance. A cache memory is the high speed accessible hardware 1

2 component to decrease the influence of the RAM access. That is, the optimized data structures determines the performance of the flow classification by decreasing the number of accesses to RAM. can be decreased significantly with the optimized data structures. In this paper, we propose a new flow classification algorithm capable of following the packet forwarding rate of the high speed routers by using the commercially available CPU, RAM, and cache memories. Then, through our experiments, we show the effects of the cache memory structures on the processing rate of flow classification. In this paper, we first describe the flow classifier and the memory requirement to store the flow statistics in Section 2. We next show the brief description of the cache memory structure and the effect of parameters of the cache memory in Section 3. We then examine the proposed flow classification scheme in Section 4. Section 5 shows our experimental results. Finally, we conclude our work with future research topics in Section 6. 2 Definition of Flow Classifier In this section, we define our flow classifier based on the statistical analysis of the flows gathered from the operating network. Requirement on the amount of the memory to store the flows is also examined. 2.1 Flow Classifier It is adequate to consider the sequence of packets belonging to the same connection as the same flow. Henceforth, it might be adequate to identify the flow using the following fields in the IP packet. {Src IP, Dest IP, Src Port, Dest Port, Protocol} However, the total length of the above fields reaches 104 bits, which is too enormous for the size of lookup table. Fortunately, we can disregard the Protocol field for flow classification since the it shows the kind of the IP packets such as TCP, UDP, ICMP, and RAW IP, and few applications have two or more Protocol values. Furthermore, HTTP, which is used by a currently dominant application WWW, uses multiple connections for each text and/or image files. Those connections can be treated as an aggregated single flow. It is even necessary when considering the fairness between users. We therefore use following fields as the flow classifier. {Src IP, Dest IP, Port Number } Note that Port Number is the lower value between the value of Src Port and Dest Port. It is because the most applications use the administrative port number smaller than 1024 that the application user cannot use. 2.2 Requirements for Storing the Flow Values The flow information to store into the router memory depends on service policy to be applied. In this section, we examine some important applications of the policy service, and evaluate the required memory size to store the

3 value associated with the flow for each service (which we will call flow value for short). Then, we will conclude that 2 bytes of memory is sufficient for each flow. Random Early Detection Based on Flow Classifications A RED (Random Early Detection) [3] mechanism is effective for the router to fairly treat the connections. In FIFO (First In First Out) or Drop Tail queueing, the sequence of packets from the particular connection tends to be discarded once the router becomes congested. In this sense, the fairness between connections cannot be established in the FIFO router. On the other hand, the RED router drops the incoming packet randomly with some probability as the queue length exceeds the pre defined threshold value. As a result, bursty packet losses can be avoided, which leads to a fair service among connections to some extent at congestion occurrence. A problem is that RED can only treat the fairness among connections, not among applications. If the flow classification is provided in an efficient way, the fairness among applications can also be established by the flow based RED [4]. In the flow based RED, the threshold is defined for each flow. When the packet arrives at the flow based RED router, the router classifies the flow, and checks the threshold for that flow. If the queue length exceeds the threshold, the router drops the packet with some probability. To store the threshold value into the flow values, it should be described as the queue length. When the flow value has 2 bytes memory, the threshold can cover 64 KB up to 256 KB by shifting 2 bit. Application to Core-Stateless Routers A core stateless router [5] is intended to fairly allocate bandwidth among flows. In CSFQ (Core Stateless Fair Queueing), the router has the per flow state, and updates the state when the packet of the flow arrives at the router. The state is described by the estimated rate of the flow in CSFQ. Since the CSFQ realizes the fairness among active flows sharing the link bandwidth, the estimated rate of the flow is normalized as (bandwidth / the total number of active flows). The router can estimate the rate of the flow when the total number of flows is less than 65,536 if each state has 2 byte memory. Fairness with the Deficit Round Robin A DRR (Deficit Round Robin) [6] algorithm can process weighted per flow queueing to provide the various kind of fairness between flows. DRR uses the deficit counter to keep the maximum number of bytes to be processed for the flow when the flow s turn comes around. When the deficit counter has 2 byte, the counter can describe up to 64 KB which is the maximum size of IP packets.

4 3 Characteristics of the Cache Memory Before describing our flow classification algorithm, we show the brief description on the cache structure to investigate the effects of parameters of the cache memory. 3.1 The structure of the cache memory Recently, a two level cache structure is widely used. It has two types of cache memory. The L1 (Level 1) cache is high speed memory and its speed is closest to CPU. The L2 (Level 2) cache is located between L1 cache and RAM, and its speed is also intermediate between L1 cache and RAM. A memory access from CPU is first reached at L1 cache to examine whether the address is cached or not. If it is cached, CPU reads from the L1 cache. Otherwise, the L2 cache is checked. If the address is not cached in the L2 cache, The address content is read from RAM. Note that more detailed caching mechanism (such as write through or write once) may affect the performance of flow classification, but it is out of scope in this paper. We will show results on performance experiments in Section 5 assuming Intel Pentium II. 3.2 Effect of Parameters Here, we examine three parameters of the cache memory; the cache block size, the degree of set associative mapping, and the cache size. Numerical results will be shown in the next section by changing those parameters Effect of the Cache Block Size The cache block is a unit of data transfer between L1, L2, and RAM. When the cache miss occurs, the cache memory reads the block of the memory. If the block size is large, the continuous address can be cached. It is therefore expected to increase the cache hit rate. However, the larger block size leads to larger memory read delay (= Block Size / Bus Width Memory Access Delay) Effect of the Set Associative Mapping Set associative mapping is recently used to increase the utilization of the cache memory. When the associativity is set to be large, the cache memory is highly and impartially utilized. However, the cost of the large associativity may be too expensive Effect of the Cache Size It is necessary to increase the capacity of the cache memory to increase the cache hit rate. However, as the cache size becomes large, the cost of the cache memory increases exponentially. Too large cache memory does not necessarily improve the cache hit ratio. It is therefore necessary to investigate the appropriate cache size for classifying flows efficiently in our context.

5 Frequency Number of Addresses Figure 1: The Number of Lower 16 bit Addresses 4 Flow Lookup Procedures 4.1 Partitioning the Flow Classifier The delay of accessing to memory is a dominant factor for determining the performance of flow classifications. Namely, reduction in the number of memory references can significantly speed up processing of packet classifications. Of course, the delay of the flow lookup would be minimized in one memory access by preparing a huge lookup table. The size of such a table, however, reaches bytes, and the large portion of the table would never be accessed. Because there is a trade off between the number of memory references and the required table size, we need to decide the structure of the flow lookup table by carefully examining both the memory size and the speed of flow classifications. We now demonstrate the sparse nature of the IP packet addresses by using the packet traced data. We gathered the traced data by the traffic monitor OC3MON [7, 8]. The monitor is placed at the gateway of Osaka University, i.e., our results reflect the characteristics of the traffic between the Internet backbone and the large scaled campus network. The total number of packets is over 27 millions. Figure 1 plots frequency distribution for the number of lower 16 bit addresses associated with the upper 16 bit address. More specifically, we plot the figure as follows. for some 32 bit IP address, we divide it into two parts; upper and lower 16 bit addresses. Then, we count the number of different lower 16 bit addresses which appear for some upper 16 bit address in the traced data. For example, there were 10 IP addresses, each of which shows 21 different lower 16 bit addresses in the traced data. From the figure, it is apparent that it follows the Zipf s law with parameter α =1[9]. There were over 40,000 addresses for lower 16 bit of addresses which did not appear in the traced data. The ratio of such addresses is about 60%. Moreover, the maximum number of lower 16 bit addresses was 97, which is less than 1% of the 16 bit address space. In summary, the flow classifier should be divided into several parts to improve memory utilization even though the number of memory references increases to some degree. In our scheme, we divide the flow classifier into following three parts; the upper 16 bit of the destination IP address, the lower 16 bit value of the destination IP address, and the source IP address + port number.

6 Frequency Number of Addresses Figure 2: Varieties of (Source IP, Port) Fields Categorized by the Destination IP Dest. IP Address bits 64K Entries 256 Entries Hash (Dest IP(17..32)) Destination Address Lookup Table Source Address Lookup Table Hash(Source IP) 1K Entries Hash ID Flow Stat Figure 3: Proposed Data Structure 4.2 Table Compression with the Hash Function As described above, the maximum number of lower addresses is 97, which is less than 1% of the 16 bit address space. Since the observation is based on our experimental traced data, it can not be always applied to other cases. However, it is also a natural assumption for assigning the IP addresses within the local administrative organization. Once such an assumption is admitted, it is useful to compress the size of the lookup table with the hash table to obtain higher utilization of the lookup table. It becomes helpful especially when the number of active entries is quite smaller than the number of total entries in the table. From the result of the previous subsection, the hash table with 256 entries is enough to store all lower 16 bit addresses. Because the number of entries is reduced from 65,536 to 256, different lower addresses may have the same hash index. Thus, we need the following procedure. When the entry is already used by another lower address, the algorithm checks the next entry of the hash table. A step by step check can reduce the number of memory accesses because the CPU caching is performed for each block. We next show the number of varieties of (Source IP address, Port Number) fields categorized by the destination IP address in Figure 2. The figure shows that about 1200 entries hash table is sufficient. 4.3 Algorithm of Flow Classification and Lookup We now describe our flow classification algorithm and lookup procedure. Figure 3 shows the data structure of our classification algorithm. The structure of the lookup table consists of four major part.

7 32 Bytes Block 7 Entries Destination IP Addr Hash ID Flow Stat Figure 4: Detailed Description of the Third Table 1. Upper 16 bit destination IP address lookup table It is a directly accessible table (65,536 entries), and each entry contains a pointer to the second table. 2. Lower 16 bit destination IP address lookup table: It is a 256 entry hash table, and each entry contains a pointer to the third table. 3. Source address lookup table: The source address consists of the combination of (Source IP address, Port Number), and it is a hash table having 1,120 entries. Each entry has Destination IP address for verification A hash value of (Source IP address, Port Number) The value associated with the flow Figure 4 illustrates the data structure of the third lookup table for the case where when the block size of the cache memory is 32 byte. To reduce the number of memory accesses, the first 4 byte is used to store the destination IP address. It is used for checking whether the destination address in the packet is equal to the stored value or not. If the verification succeeds, no memory reference is performed because the flow value is cached. In this structure, (32 4)/4 =7entries can be stored into one cache block. The required number of blocks becomes 160 to create the hash table with 1,120 entries. We finally show the address calculation algorithm for the flow lookup. 1. Ptr1 = HeadTable[Dest Addr(1..16)] 2. Ptr2 = Ptr1[Hash(Dest Addr (17..32))] 3. stateptr = Ptr (Hash(Src, Port) / 7) + (Hash(Src, Port) mod 7) Numerical Experiments and Discussions In this section, we show our experimental results on the flow lookup delay and the effect of parameters in the cache memory.

8 5.1 Experimental Setup We implemented our proposed scheme on the following architecture. Intel Pentium II 450 MHz 16 KB 4 way set associative L1 cache 512 KB 4 way set associative L2 cache 256 MB memory Linux GNU gcc version (release) We again use the packet traced data that we have described in Section 2. To investigate the effect of parameters of the cache memory, we used trace driven cache simulator, Dinero IV version 7 [10]. Moreover, we use RDTSC [11] instruction in Pentium II to obtain the number of CPU cycles for the flow classification. Dinero IV reports the total number of memory references N, and the cache miss ratio of L1 and L2 (denoted by M L1 and M L2, respectively). We also obtained total number of packets P, the reference delays of memory, L1, and L2 cache (N Mem, N L1, and N L2, respectively). We can determine the mean access delay for the flow classification as; mean access delay = N/P (1 M L1 )N L1 (1) M L1 (1 M L2 )N L2 M L1 M L2 N Mem We finally note that in an actual situation some update procedure is necessary according to the adopted policy, but we neglect it in the following results. 5.2 Effects of Cache Memory Sizes We first show the effect of the L1 cache size on the cache miss ratio in Figure 5 by utilizing the cache simulator Dinero IV. The cache miss ratio can be decreased by the increased cache memory size. However, when the size is beyond 64 KB, we cannot observe the significant improvement by the cache size. Thus, the currently available L1 cache size of Intel Pentium II (and III) is slightly smaller than the expected one. Figure 6 shows the relation between the associativity and the cache miss ratio. The same tendency can be found in the figure. The four way set associativity of Intel Pentium is slightly smaller. Due to space limitation, we cannot show results on the L2 cache, but its effect follows the same tendency as the L1 cache. The reasonable values were 256 KB for the cache size, and four for the associativity.

9 Cache Miss Rate Cache Size Figure 5: Effect of the Cache Size (L1) Cache Miss Rate Associativities Figure 6: Effect of the Associativity (L1) 5.3 CPU Cycles and Required Memory Size We finally show the average CPU cycles necessary for the packet classification by using RDTSC in Table 1. The values of sizes of the L1 and L2 caches and associativities are based on Intel Pentium II, and are fixed as described before. The table shows that our implementation can classify about 2 million packets per second. If we assume the average packet size is 552 Bytes [7], our scheme can follow up to 10 Gbps of the switching capacity or the link speed. The required memory size was 49.1 MB. It is because the lookup table size is fixed for each flow in the current implementation. If we can change the size of lookup table dependent on the flow, the required memory size could be reduced. which is the future research topic. 6 Concluding Remarks In this paper, we have proposed a new packet classification algorithm capable of following the packet forwarding rate of the high speed routers with the commercially available CPU, RAM, and cache memories. Through our experiments, we have shown the effects of parameters of the cache memory on the processing rate of packet Table 1: CPU Cycles CPU Cycles Processing Delay Packet Lookup Rate nsec MPPS

10 classification. For future research topics, we need to evaluate our implementation with the public domain trace archives to verify the generality of our scheme. References [1] Tzi-cker Chiueh and P. Pradhan, High performance IP routing table lookup using CPU caching, in Proceedings of IEEE INFOCOM 99, April [2] T. V. Lakshman and D. Stiliadis, High speed policy based packet forwarding using efficient multi dimensional range matching, in Proceedings of ACM SIGCOMM 98, pp , September [3] S. Floyd and V. Jacobson, Random early detection for congestion avoidance, IEEE/ACM Transactions on Networking, pp , July [4] D. Lin and R. Morris, Dynamics of random early detection, in Proceedings of ACM SIGCOMM 97, pp , September [5] I. Stoica, S. Schenker, and H. Zhang, Core stateless fair queueing: Achieving approximately fair bandwidth allocations in high speed networks, in Proceedings of ACM SIGCOMM 98, pp , September [6] M. Shreedhar and G. Varghese, Efficient fair queueing using deflicit round robin, IEEE/ACM Transactions on Networking, vol. 4, pp , June [7] K. Thompson, G. J. Miller, and R. Wilder, Wide area Internet traffic patterns and characteristics, IEEE Network, pp , November [8] S. Ata, M. Murata, and H. Miyahara, Analysis of network traffic and its application to design of high speed routers, to appear in IEICE Transactions on Information and Systems (D-I), May [9] M. Aida, N. Takahashi, and T. Abe, A proposal of dual Zipfian model for describing HTTP access trends and its application to address cache design, Transactions of IEICE, vol. E81 B, pp , July [10] J. Edler and M. D. Hill, Dinero IV : Trace-driven uniprocessor cache simulator, available at markhill/dineroiv, [11] Intel Corporation, Using the RDTSC instruction for performance monitoring, Pentium II Application Note, available at HTM, 1998.