CRC32 Based Signature Generation Methods for URL Routing

CRC32 Based Signature Generation Methods for URL Routing Zornitza Genova Prodanoff and Ronnie King Department of Computer and Information Sciences University of North Florida 4567 Saint Johns Bluff Road Jacksonville, Florida 32224 {zprodano, rking}@ unf.edu Keywords: CDNs, URL routers, HTTP redirection, NDIS drivers. ABSTRACT Signature algorithms have already been proposed for use in URL routers. Cyclic Redundancy Check (CRC) codes can be used as URL signatures. When a packet is received, CRCs are generated on the fly with the use of a very simple hardware circuits. Existing network adapters are not programmed to generate CRCs on packet subfields. However, it is possible to use the single CRC circuit of a network adapter to simultaneously generate the packet CRC and CRCs on any subfield (such as the URL string in the HTTP GET request header) of the packet. The available from the CRC circuit intermediate CRC codes can then be used to produce a URL signature for each incoming HTTP request and do URL routing table look-up. In this paper we evaluate alternatives for URL signature generation that rely on hardware or software computation of CRC32 codes. Our results indicate that the delay of producing the signature in software is smaller (100s of nanoseconds) as compared to the alternative of using the CRC circuit in existing adapters (10s of microseconds). 1. INTRODUCTION Content Distribution Networks (CDNs) allow for a single document or service to be contained in multiple servers and caches throughout the Internet. A key challenge is how to select the best content source for a given request. The criteria for a best source include minimizing network load while satisfying a response time requirement for the client. Typically, the goal is for a client to access the nearest copy of a document or service. Client based methods (e.g., [1]) use a ping type approach to find the fastest responding server for a given request. These methods generate excessive and redundant overhead traffic. Anycasting and centralized approaches have been studied ([5],[13]) as well, where routers or specialized services determine the best server. These methods are high in overhead or require infrastructure changes. In Internet caching systems, sharing of URL lists enables routing or forwarding of requests between the cache sites ([4],[12]). Figure 1 shows a CDN where a given document can be found in its origin server s directory, in a temporary server (e.g., via a distribution service that shares servers on demand), and/or in an Internet cache structure. In the Web, a variable length URL identifies a document or service. It may be desirable that a single URL map to multiple content sources with the best source containing the document that is sent to the client. For a given site, the number of URLs is typically very large (1000 s to 100,000 s) and their existence can be transient (e.g., due to aging-out of documents in caches). This is in contrast to IP addresses, which are typically few and fixed for a given site. In a URL router [18], an HTTP request resolves into an IP address based on the known state of URL locations at content sources. For each such request the processing is performed at the application (or HTTP aware communications layer). The URL string from each incoming request is examined and a look up into a URL based routing table is then performed. The URL routers are front-end nodes to edge-based servers and caches. Sharing of URL lists, along with knowledge of network and server state, is necessary in order to build URL routing tables in the URL routers. Client Sharing of URL lists IP switch HTTP request to the best source URL switch router Figure 1. A CDN Origin server Caches Temporary server

Given the very large number of URLs and their rate of change, methods have been investigated of compression for sharing the lists of URLs contained at content sources. These compressed lists of URLs are called URL digests. Routing tables for URL routers [6] can be built based on the information communicated in the digests. URL routers need routing tables similar to IP routing tables in IP routers. However, the methods used for IP routing cannot scale to URL routing. In URL routing the number of destination addresses (URLs) greatly exceeds the number of IP addresses. In addition, URLs are not stable new ones are constantly created and old ones disappear. The large size of URLs, their very large number, and their dynamic nature makes URL routing a difficult problem. Signature algorithms have already been proposed for use in Web routers [xx]. The use of signatures can help both reduce size of digests and improve look-up performance of URL routing tables in URL routers. Cyclic Redundancy Check (CRC) codes can be used to produce signatures for incoming URLs when a packet is received. These signatures are generated on the fly and then being used to do routing table look-up. This will enable the use of greatly reduced in size routing tables, where each URL is represented by its signature. This reduction is caused by substituting a variable length string, about 50 byte long on an average [x10challenges], with a 32 bit fixed length signature value. A signature algorithm using CRC32 codes will result in 2 32 possible signatures, providing a sufficiently large signature space for most server directory sizes. There are two general alternatives for on the fly signature generation. In the case of a one armed URL router [6], the network adapter can generate the CRC or the CRC can be generated by software. The hardware required to generate a CRC is a serial shift register and XOR circuit and is an integral part of all network adapters. At the URL router, all incoming HTTP requests have their URLs serially encoded into a CRC. Existing network adapters are not designed to generate CRCs on subfields within a received packet. However, it is possible to use the single CRC circuit of a network adapter to simultaneously generate the packet CRC and CRCs on any subfields of the packet if partial CRC values can be accessed. This study evaluate the performance of the two general alternatives for generating CRC32 signatures for the purposes of URL routing software vs. hardware generation. The remainder of the paper is organized as follows. Section 2 provides a brief overview of hardware computation of CRC, and Section 3 describes a previously introduced idea to compute CRC using existing network adapters and intermediate CRC values for packet subfields. Section 4 presents an evaluation and comparison of hardware vs. software generation of CRC32 signatures as used by URL router applications. The last section Section 5 presents a brief Summary and outlines Future work. 2. HARDWARE GENERATION OF CRC Cyclic Redundancy Check (CRC) codes have been widely used for nearly 40 years in communication networks. CRCs introduce control redundancy into the data in order to provide error detection. Over the years the implementation methods used to generate CRC have constantly changed. The motivation for these changes has been driven by the need to support transmission speeds on a cost-sensitive market for Network Terminal (NT) hardware [11]. The basic idea behind CRC is simply to use the remainder generated by dividing one polynomial by another. For example, Ethernet uses a well-known divisor polynomial D(x) of degree 32 to allow a sender to view their data as a polynomial P(x) that is transmitted over a link alone with a 32- bit remainder that comes from dividing P(x) by D(x). So once a sender and receiver agree on a divisor polynomial, the CRC algorithm can be implemented in either software or hardware. However, the obvious drawback to a software solution is speed compared to a dedicated hardware solution. Traditional hardware implementations rely on the fact that the remainder is just the bitwise exclusive-or of the polynomials P(x) and D(x). As a result, the CRC algorithm is easily implemented in hardware using a linear shift register and a few XOR gates attached to a serial data stream. The number of bits in the shift register equals the degree of the divisor polynomial D(x), which is 32 bits in the case of Ethernet. The number of XOR gates and their positions is determined by D(x). If there is a term x n in the divisor polynomial then an XOR gate would be needed in front of bit n in the shift register, assuming the bits in the register are numbered zero to one less then the degree of the divisor polynomial. Figure 2 shows the traditional hardware used to produce the CRC for the divisor polynomial given by x 3 + x 2 + x 0. Serial Data Stream P(x) XOR gate D(x 0 ) Traditional Hardware CRC Calculation Using Linear Shift Register x 0 x 1 x 2 XOR gate D(x 2 ) The XOR gates position depends on the terms in the divisor polynomial and is one less then the number of terms. D(x) = x 3 + x 2 + x 0 Figure 2. An r-bit Feedback Shift Register [17]

Traditional implementations have two main problems. First, it is difficult to keep pace with today s high speed communication systems while using sequential hardware designed to compute CRCs. Second, it is not possible to reconfigure fixed logic to support a need to change the divisor polynomial. To overcome the speed limitations of a linear shift register a parallel solution has been introduced. The speed-up factor is over four times faster then using a traditional linear shift register approach [15]. Existing network adapters are not programmed to generate CRCs on subfields within a packet such as the URL field. However, the problem of generating CRCs for subfields within a packet can be solved by allowing the intermediate values form the packet CRC calculation to be accessed later as elaborated on in the next section. 3. PACKET CRC AND CRC32 URL SIGNATURES A single CRC32 circuit can be used to simultaneously generate an overall packet CRC32 and a CRC32 code for a subfield of the packet. The idea of using the CRC value of subfields within a packet to reduce the time to recalculate the packet CRC was introduced in [6]. This concept allows packet CRCs to be recomputed many times within a network at a low cost. A packet CRC is being recomputed on various network hops. For example, virtual circuit switching type protocols such as Multiprotocol Label Switching (MPLS) or ATM, often have to recalculate CRC values at each output port. Any node that receives a packet usually needs to swap the virtual circuit identifier (VCI) with a different value. Since the VCI is embedded within the payload, the CRC must be checked and recomputed at each hop in the network. Solutions to the problem of how to recalculate the CRC of a packet with an altered field are given in [10],[3], and [7]. The main idea in these solutions is to combine the CRC from the original (unaltered packet) and the CRC of the altered section of the packet. By applying a property introduced in [3] the packet CRC can be updated after a computation over the altered fields: f1 + f 2 f1 f 2 Rem = Rem + Re m (1) P P P Where f 1 and f 2 are shifted message polynomials and P is the CRC generator polynomial. This property becomes clear once we take a look at an example. Suppose the generator 3 2 polynomial P ( x) = x + x + 1, in which case the polynomial would be of degree three. Now consider the message 7 4 3 1 m ( x) = x + x + x + x (i.e., 10011010). Let m 1 represents the first four bits of the message and m 2 the last four bits, i.e. 7 6 5 4 m 1 represents the message polynomial x + 0x + 0x + x 3 2 1 (10010000), while m 2 equals x + 0x + x (1010). To produce the shifted message polynomial f 1, we multiply m 1 by 3 x since the degree of our generator polynomial is three, this step will append three zeros on the right side of m 1 (10010000000). The same technique is used to produce f 2 (1010000). Note: f 1 + f 2 equals m (x) shifted left three spaces. A simple use of polynomial arithmetic modulo 2 and long division to divide ( f 1 + f 2 ) by P(x) will confirm the CRC value (remainder) is five. A quick check will also reveal that the CRC given by dividing F1 by P(x) is four and the CRC for f is one. This result agrees with Equation (1) above. 2 If we assume the value of a network adapter CRC32 shift register can be sampled at the byte boundary of the last byte before the URL in the HTTP request and again at the last byte in the URL, then these partial CRCs can be easily combined to generate a URL signature. Let A 0 be the bit sequence from the beginning of the packet to the last byte before the URL and A 1 be the bit sequence from the beginning of the packet to the last byte of the URL. Then the CRC of the URL is given by A2 A1 A0 Rem = Rem where A 2 is the bit sequence of P P the URL field. To take advantage of the simultaneous generation of the overall packet CRC32 and a CRC32 code for a subfield, an URL router application needs to be supplied intermediate CRCs values. For this purpose there are two alternatives one option is to modify the hardware of existing interface cards to add logic to recognize HTTP layer URL string delimiters (part of the HTTP GET request header), and then scan the CRC32 circuit to extract the corresponding intermediate CRC32 values. An alternative is to modify existing miniport drivers to supply to the URL router application CRC32 values in parallel with supplying packet bits. In the evaluation, provided next in this paper, we consider the second solution, since it does not require modification of the interface card hardware. 4. EVALUATION Our testbed consisted of a 3.06GHz Dell Optiplex Pentium 4 PC with a WindowsXP Pro operating system. When implementing the subfield CRC generation, we first evaluated the delay caused by accessing an IEEE 802.3 Ethernet compliant network interface card. Hence, we first measure the I/O delay. For the purposes of this measurement, we used the winpcap set of library calls [20]. Since the URL router is HTTP aware, we conducted the measurements in user mode, but surpassing operating system services such as TCP/IP protocol logic. This application was running at the MS protocol driver layer, since miniport and intermediate drivers cannot communicate with user-mode applications (see [14] for elaboration). A possible implementation was assumed to rely on a modification (not

implemented as part of this evaluation) at the NDIS miniport driver level, where the content of the CRC32 circuit is scanned and propagated to the URL router application under a single I/O operation, bundled within the packet bits. We evaluated the performance of a hardware solution to the CRC32 signature generation for the purposes of URL routing as described in Section 3. Our results provide a pessimistic estimate of the time needed to transfer the two CRC32 signatures, at the beginning and the end of the URL string, to the URL router application, since only a single I/O operation was assumed to be sufficient for the transfer of packet bits, as well as signature bits. We measured the delay to access the network interface card and perform a raw send() of 1000 packets, each consisting of 84 bytes, including overhead bytes at all layers above the MAC layer. All the 1000 packets are submitted to the adapter via a single I/O operation. We choose this large number of bytes in order to ensure a valid granularity of timing results. On an average, it takes about 3 microseconds to transfer a single byte. Ignoring additional circuit delays, this would suggest that the delay for accessing the content of the CRC32 circuit will be at least four times that value, since the signatures are four bytes long. In order to produce a CRC32 signature with this implementation, the content of the CRC32 circuit will need to be scanned twice once at the byte boundary, where the URL strings begins, and once at the end of the URL. Hence, on an average, hardware based CRC32 generation will require a minimum delay of about 24 microseconds. We have also evaluated the generation of CRC32 in software based on an implementation used in [8],[19]. This implementation is adapted from an algorithm introduced in [16]. The algorithm uses an 8-bit table look up method of generating CRCs and is implemented in C language, using the Borland C++ 5.5.1 compiler. Timing results were produced by using the timeb library calls and data type definitions. The code used is available on-line [2]. Figures 3 and 4 show the C code for the two main parts of the algorithm: a CRC32 generation function and a table initialization function. typedef unsigned short int word16; typedef unsigned int word32; #define POLYNOMIAL 0x04c11db7L static word32 crc_table[256]; word32 update_crc(word32 crc_accum, byte *data_blk_ptr, word32 data_blk_size){ register word32 i, j; for (j=0; j<data_blk_size; j++){ i=((int)(crc_accum>>24)^*data_blk_ptr++)&0xff; crc_accum = (crc_accum << 8) ^ crc_table[i];} crc_accum = ~crc_accum; return crc_accum; } Figure 3. CRC32 Code Generation typedef unsigned short int word16; typedef unsigned int word32; #define POLYNOMIAL 0x04c11db7L static word32 crc_table[256]; void gen_crc_table(void){ register word16 i, j; register word32 crc_accum; for (i=0; i<256; i++){ crc_accum = ( (word32) i << 24 ); for ( j = 0; j < 8; j++ ){ if ( crc_accum & 0x80000000L ) crc_accum = (crc_accum << 1) ^ POLYNOMIAL; else crc_accum = (crc_accum << 1); } crc_table[i] = crc_accum; } } Figure 4. CRC32 Table Initialization We have produced CRC32 signatures for a total of 254,727 unique URL strings. These URLs were extracted from HTTP trace from a NASA Web server [9]. The original number of URLs in this trace was a total of 1,569,898. To achieve the expected level of precision and obtain valid timing results, for each URL string there were a total of 1,000,000 CRC32 calculations produced at each iteration of the CRC generation function. The results indicate that on an average it would take about 236 nsec to generate a CRC32 signature for a single URL. These results show that a software generation of CRC32 signatures as used for the purposes of URL routing can result in several magnitudes lower delays as compared to methods using the CRC32 circuits within existing network adapters. 5. SUMMARY AND FUTURE WORK In this paper we have shown that the software generation of CRC32 signatures, as used for table look up in URL routers, results in smaller delays as compared to using a hardware based implementation that takes advantage of existing network adapter logic. Future work will consist of problems related to the scaling of Web services from local server clusters to globally distributed CDNs with automatic server selection. For example, we plan to address the application of signature methods to high-speed URL switching suitable for 10-Gigabit Ethernet networks. However, the applications of signatures to switching are not limited to routing in CDNs. Future work may include the use of signature methods for routing table look-up in IPv6 routing. REFERENCES [1] R. L. C. Carter, M.E., "Dynamic server selection in the Internet," presented at Third IEEE Workshop on the Architecture and Implementation of High Performance Communication Subsystems (HPCS95), 1995. [2] K.J. Christensen, Tools Page, URL: http://www.csee.usf.edu/~christen/tools/toolpage.html

[3] S. Dravida, Error Control Aspects of High Speed Networks, Proceedings of IEEE INFOCOM 92, pp. 272-281, 1992 [4] L. C. Fan, P.; Almeida, J.; Broder, A.Z., "Summary Cache: a scalable wide-area Web cache sharing protocol," presented at SIGCOMM'98 Conference. Applications, Technologies, Architectures, and Protocols for Computer Communication, Vancouver, BC, Canada, 1998. [5] Z.-M. B. Fei, S.; Zegura, E.W.; Ammar, M.H., "A novel server selection technique for improving the response time of a replicated service," presented at IEEE INFOCOM '98, the Conference on Computer Communications. Seventeenth Annual Joint Conference of the IEEE Computer and Communications Societies., San Francisco, CA, USA, 1998. [6] Z. Genova and K. Christensen, Using Signatures to Improve URL Routing, Proceedings of IEEE International Performance, Computing, and Communications Conference, pp. 45-52, April 2002. [7] M. Gutman, A Method for Updating a Cyclic Redundancy Code, IEEE Transactions on Communications, Vol. 40, No. 6, pp. 989-991, June 1992 [8] C. M. Heard, Implementation of CRC32 in Software. URL: http://cell-relay.indiana.edu/cellrelay/publications/software/crc [9] HTTP requests to the NASA Web server, URL: ftp://ita.ee.lbl.gov/traces/. [10] D. Irvin, Preserving the Integrity of Cyclic-Redundancy Checks when Fully Protected Text is intentionally Altered, IBM Journal of Research and Development, Vol. 33, No. 6, pp. 618-626, November 1996. [11] J. Kadambi et al, Gigabit Ethernet, Prentice Hall PRT, ISBN 0-13-913286-4, 1998. [12] B. S. N. Michel, K.; Reiher, P.; Lixia Zhang, "URL forwarding and compression in adaptive Web caching," presented at IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies, Tel Aviv, Israel, 2000. [13] K. Moore, "SONAR: A network proximity service version 1," 1998. [14] MSDN Library, Network Devices and Protocols: Network Drivers, URL: http://msdn.microsoft.com/library/default.asp?url=/library/ en-us/network/hh/network/101rm_0odj.asp. [15] T. B. Pei and C. Zukowski, High-speed parallel CRC cicuits in VLSI, IEEE Transaction Communication, Vol. 40, No. 4, pp. 653-657, 1992. [16] A. Perez, "Byte-wise CRC Calculations," IEEE Micro 3, 40 (1983). [17] T.R.N. Rao and E. Fujiwara, Error-Control Coding for Computer Systems, Prentice-Hall, 1989. [18] A. Roginsky, K. Christensen, and S. Polge, Efficient Computation of Packet CRC from Partial CRCs with Application to the Cells-in-Frames Protocol, Computer Communications, Vol. 21, No. 7, pp. 653-661, June 1998. [19] D. Sarwate, Computation of Cyclic Redundancy Checks via Look-up Table, Communications of the ACM, Vol. 31, No. 8, pp. 1008-1013, August 1988. [20] WinPcap: the Free Packet Capture Architecture for Windows, Version 3.0, URL: http://winpcap.polito.it/. BIOGRAPHIES Zornitza Genova Prodanoff is currently an Assistant Professor in the Department of Computer and Information Sciences at the University of North Florida. She holds a PhD in Computer Science and Engineering from the University of South Florida, an MS in Computer Science, and a BA in Information Systems Management. She is the author, and coauthor of several book chapters, journal articles, and conference papers on topics including performance evaluation of computer networks, network programming, time-constrained multimedia presentations, and database design. Her professional affiliations include IEEE, IEEE Women in Engineering, ACM, and ASEE. Ron King is currently an Assistant Professor at the University of North Florida in the College of Computing, Engineering, and Construction. The University of South Florida awarded Dr. King a doctorate in 1997 for his doctoral dissertation entitled Modular Design for Large Broadband ATM Switching Fabric. In 1997, Dr. King took a position with the Defense Information Systems Agency as senior technical advisor within the Center for Computer Systems Engineering, Fall Church, Virginia, supporting various Department of Defense information systems, including the White House Communications.