Testing the Performance of Multiple TCP/IP Stacks

Transcription

1 Testing the Performance of Multiple TCP/IP Stacks White Paper by John L. Wood, Christopher D. Selvaggi, & John Q. Walker Contents Introduction... 2 Our Implementation... 2 TCP and UDP Performance... 3 Stack Performance... 7 Performance Implications of the TCP Close Conclusions Appendix A: Test Methodology14 Appendix B: Details of Test Scripts Copyright Information Performance of TCP/IP applications varies widely among stacks, operating systems, and hardware platforms. We describe our experiences with TCP and UDP performance on eight operating systems, ranging from 3.11 to UNIX. We address issues such as performance differences between TCP and UDP, throughput and response time differences among different operating systems and Ethernet speeds, and how a program s use of Sockets calls can affect performance. An earlier version of this paper was presented at the Computer Measurement Group in 1997: John L. Wood, Christopher D. Selvaggi, and John Q. Walker II, Testing the Performance of Multiple TCP/IP Stacks, Proceedings of CMG97, December 7-12, 1997, volume 1, pages Testing the Performance of Multiple TCP/IP Stacks 1

2 Introduction Computer networks change at an ever-greater rate. Software vendors package TCP/IP protocol stacks with their operating systems, and new applications, including browsers and , readily make use of these stacks. Network administrators upgrade the hardware and software in their networks as users find they need ever more bandwidth. But, how does network hardware and software really perform? What happens when it is lightly loaded, and what happens when it reaches its boundaries? Which network hardware has the capacity to handle traffic in a given network? We used a set of software programs to drive the TCP and UDP protocol stacks, measuring the round-trip performance of three typical network transactions. We describe here the differences we saw between TCP and UDP. We compare the performance of protocol stacks among a variety of operating systems, with different Ethernet speeds. We discuss what we learned about tuning the stacks. We also discuss in detail the use of the Sockets close, and how it affects the performance of short, back-to-back connections. Our Implementation We used a software application named Chariot to explore these basic network performance questions. Chariot generates network traffic between pairs of computers and observes the performance of the traffic. Traffic patterns are highly tailorable, letting you recreate the traffic generated by real user applications. Performance tests, capturing everything about the traffic patterns and the computers involved, can be saved and reliably repeated. For example, you can see the effect of changing the hardware or software along network paths; you can see the effect of adding new users; or you can track the level of service available in the network. Console Endpoint 1 Endpoint 2 Figure 1: Performance tests are set up at the Console and run between a pair of computers, labeled Endpoint 1 and Endpoint 2. The set of programs is operated from the console, where you create and run tests. Creating a test consists of deciding which computers to use and the type of data traffic to run between them. We refer to the computers executing the tests as endpoints. An endpoint pair comprises the network addresses of the two endpoint computers, the network protocol to use between them, and the type of application to emulate. For each endpoint pair, select an application script corresponding to the application you are simulating. The endpoint programs use an application script to generate the same data flows that an application would, without installing the application. A set of pre-built application scripts provides standard performance benchmarks and emulates common end-user applications. Today, endpoints run on twenty operating systems 1, supporting six network protocols 2. Support for TCP and UDP is available on all these systems. On the platforms of 1997, endpoints issued calls to the WinSock application programming interface (API) at the 1.1 level. On the other platforms, endpoints issue Sockets calls. Our programs issued their WinSock and Sockets calls as blocking calls. We saw a performance degradation of 20% to 50% when using nonblocking calls on the code paths where we were measuring performance. Chariot supports tests with multiple concurrent connections between any endpoints. We limited the scope of our testing here to just one connection at a time between endpoints. 1 Operating systems tested: NT, 95, and 3.11, OS/2, NetWare, AIX, HP-UX, and Sun Solaris. 2 Network protocols supported: APPC, IPX, SPX, TCP, UDP, and RTP. 2 Copyright NetIQ Coporation,

3 We used three benchmark scripts in our testing. The application scripts called CREDITS and CREDITL simulate repeated credit-check transactions: an endpoint sends a small record and gets an acknowledgment. Latency and turnaround time have a big effect when running these scripts; buffer sizes have little effect. The FILESNDL application script simulates a file transfer, by sending a large block of data and getting an acknowledgment in return. This causes multiple, full buffers to flow on the network in one direction, so the stacks buffering and windowing has a greater effect than the latency and turnaround time. CREDITS (credit-check, short connections) a transaction consists of sending 100 bytes from the first endpoint to the second, which replies with a one-byte acknowledgment. A connection between the endpoints is brought up and taken down within each repeated transaction; the connection time is measured as part of each transaction. CREDITL (credit-check, long connection) as with CREDITS, 100 bytes is sent and one byte is received in reply. One connection is brought up and multiple transactions are repeated before taking it down. In comparing CREDITL to CREDITS, you can see the effect of bringing up and taking down connections. FILESNDL (file-send, long connection) a transaction consists of sending a large number of bytes and receiving a one-byte confirmation. As with CREDITL, connections span many transactions and connection time is not measured. These three benchmark scripts and the parameters we used are described in further detail in Appendix B. We used a matched pair of Intel-based computers with matched sets of Ethernet adapters when testing with the PC operating systems. With disk partitioning and the multi-boot program System Commander, we configured five operating systems on each of the two computers. We also tested from one of these computers (running NT as Endpoint 1) to three different UNIX platforms. The setup and configuration for the computers are described in Appendix A. TCP and UDP Performance Programs that communicate over a network follow a protocol for the exchange of data. A connectionoriented or stream protocol provides reliable delivery of data at the cost of a set of initialization and termination procedures. TCP is a connectionoriented protocol. A connection-less or datagram protocol, such as UDP, provides a best-effort delivery service. The network tries to deliver application data to the recipient, but if there are problems along the way, the data is lost. Moreover, the application is not notified of the loss. In spite of the unreliable nature of datagram protocols, they are frequently used by network applications because they do not incur the overhead associated with connection establishment and takedown. UDP Reliable Datagram Implementation Datagrams work like two people exchanging letters using the postal service: there is no guarantee letters arrive in order or even at all. If they are to use UDP, those applications must follow an approach that ensures the data is properly exchanged. Such an approach typically requires the use of: acknowledgments, to let the sender know the partner has received data, timers, so the sender can retransmit its data if it doesn t receive an acknowledgment from the partner soon enough, and a flow control mechanism, to prevent the sender from flooding its partner with too much data. Our test programs employ a straightforward datagram protocol. 1. A window scheme is used as the flow control mechanism: a sender transmits a certain amount of data before waiting for an acknowledgment from the receiver. 2. A sender waits for a period of time (the retransmission time-out period) to receive an acknowledgment from the partner. If the acknowledgment does not arrive in time, the Testing the Performance of Multiple TCP/IP Stacks 3

4 sender retransmits the window of unacknowledged data. 3. An acknowledgment is sent by a receiver when a window is filled or all the data is received for a script RECEIVE command. 4. In the case where the receiver can detect lost datagrams, it sends an acknowledgment indicating how much of the window was received in sequence, allowing the sender to retransmit what was not received. This datagram protocol is a subset of the functionality TCP provides to ensure that data is received reliably. If an application sends one traffic flow pattern all the time, the reliable receive algorithms can be tuned so that UDP will outperform TCP. As the traffic flow pattern becomes more varied, it is hard to build a reliable transport mechanism on UDP that outperforms the reliable transport mechanism implemented by TCP. It is not surprising that TCP performed as well as UDP or better in many of our tests. TCP and UDP provide different levels of functionality. TCP provides reliable transport, where UDP allows for low-overhead stateless transactions. UDP works best with applications which use short transactions. For long-running transactions TCP is more efficient and can overcome the connection overhead. In general, we want to demonstrate how performance is affected by protocol differences. In comparing TCP and UDP, we used one operating system and TCP/IP protocol stack: the latest version of Microsoft s NT 4.0. Our comparison involves several variables: the quality of the TCP and UDP layers in NT, Chariot s implementation of a datagram protocol, and the network layout. We used two matched computers on a single LAN segment. In these tests, we did not explore multi-hop network topologies. Similarly, we did not measure how routers treat TCP and UDP. Connection Establishment Overhead Test TCP is connection-oriented and therefore incurs setup time while the connection is established. This shows up as slower response time, for transactions where the connection is started and stopped. The cost is incurred regardless of the underlying network type. This protocol difference is demonstrated using the CREDITS and CREDITL scripts. A test consisted of one pair of computers, using 10 Mbps and 100 Mbps Ethernet, for both TCP and UDP. The results appear in the graphs below CREDIT S, TCP CREDIT L, TCP CREDIT S, UDP CREDIT L, UDP Figure 2: Response time with 10 Mbps Ethernet, using the CREDITS and CREDITL scripts. Units are shown in seconds per transaction; lower is better. 4 Copyright NetIQ Coporation,

5 CREDITS, TCP CREDITL, TCP CREDITS, UDP CREDITL, UDP Figure 3: Response time with 100 Mbps Ethernet, using the CREDITS and CREDITL scripts. Units are shown in seconds per transaction; lower is better. Connection establishment overhead clearly causes poorer response time for short-lived transactions. The response time can be improved by using longlived transactions. The response time for UDP is about the same in both the CREDITS and CREDITL scripts, as it should be, since no connection-establishment overhead cost is incurred. Using a 10x faster Ethernet link did not result in a 10x improvement in response time this was the case for both protocols. We believe computer constraints, such as CPU and bus speeds, limit what is attainable. Comparing TCP and UDP Response Time The CREDITL script also serves well as an applesto-apples test. The reported measurements do not include connection-establishment overhead, so the comparison is between how well TCP and UDP send data. The results in Figure 4 show that better response time can be had with TCP. TCP s reliable transport algorithm is able to outperform the reliable transport algorithm we implemented for UDP. When the overhead of connection setup and take down is removed, this test shows that TCP provides an efficient mechanism for achieving excellent response time Mbps, TCP 10 Mbps, UDP 100 Mbps TCP 100 Mbps, UDP Figure 4: Response time with 10 Mbps and 100 Mbps Ethernet, using the CREDITL script. Units are shown in seconds per transaction; lower is better. Testing the Performance of Multiple TCP/IP Stacks 5

6 Comparing TCP and UDP Throughput When optimizing throughput, four variables are important: file size, send buffer size, window size, and maximum transmission unit (MTU). In comparing TCP and UDP, we chose UDP and TCP values that would maximize throughput. We used the FILESNDL script to compare the throughput differences of UDP and TCP. Each test used a send size of 1,460,000 bytes and a send buffer size of 32,767 bytes for TCP and 8,863 bytes for UDP. The TCP window size was 8,760 bytes, and the UDP window size was set to 17,726. With 100 Mbps Ethernet we were able to increase the send size to 32,543 and the window size to 130,172. The MTU size was left at the maximum of 1,500 for all tests. Changing the Send Buffer Size The network packet size, sometimes called the frame size, is the maximum amount of data a network adapter can transmit at one time. When the size of a datagram exceeds the packet size, the network protocol stack breaks the datagram into pieces, each no larger than the packet size. This process is called datagram fragmentation. The process of putting the packets back together, which is done at the destination computer, is called datagram reassembly. IP supports fragmentation and reassembly of higher-layer protocols such as UDP and TCP. If the TCP/IP network protocol stack can reassemble datagram fragments faster than the application software can issue API send calls, tests can run faster -- if you configure the send buffer size as large as possible. On the other hand, a large number of datagram fragments may increase the congestion in a network and, therefore, the likelihood that one of them may be dropped. If that occurs, the entire datagram must be retransmitted, causing poorer performance. TCP avoids IP datagram fragmentation when possible by breaking data into MTU-sized pieces. Since TCP ensures the delivery of every IP datagram it sends, if one datagram is lost it only requires the retransmission of that datagram. UDP on the other hand does not avoid IP datagram fragmentation. UDP will pass on whatever size buffer it gets and then IP fragments it and sends it out. If a 32-KB send is issued to TCP, the block of data is broken up by TCP into MTU-sized pieces. If one MTU is lost, only that MTU needs to be retransmitted. If a 32-KB send is issued to UDP, the 32 KB is passed directly to IP and IP breaks it into MTU-sized pieces. If one MTU, the whole 32 KB block is considered lost and the whole 32 KB have to retransmitted. For the best performance, the TCP send buffer size should be as large as possible. For UDP throughput tests, the send buffer should be tuned for the underlying networks. We found a send buffer size of 8,863 bytes to be best for 10Mbps and 32,543 for 100Mbps. This causes IP fragmentation, but not enough to result in lost packets and thus degrade performance. 6 Copyright NetIQ Coporation,

7 Mbps, TCP 10 Mbps, UDP 100 Mbps, TCP 100 Mbps, UDP Figure 5: Throughput with UDP and TCP, on 10 Mbps and 100 Mbps Ethernet, using the FILESNDL script. Units are shown in Mbps; higher is better. When using 10 Mbps Ethernet, UDP and TCP perform roughly the same. The 100 Mbps Ethernet tests show the difference between them. TCP outperforms UDP by approximately 3 Mbps. Even a highly-tuned UDP algorithm has a hard time significantly improving the performance possible with TCP. We first tested throughput, using the FILESNDL script. We used the same operating system for both pairs (except for the UNIX tests where we used a NT computer as Endpoint 1). We ran the same set of tests on 10 Mbps and 100 Mbps Ethernet LANs. Stack Performance We used a single operating system and protocol stack when comparing TCP and UDP performance, above. To examine stack performance, we used the same hardware, but varied the operating systems and stacks. We used each operating system and stack with their shipped default parameters. The setup is described in Appendix A NT WFW OS/2 NetWare Warp Solaris 2.4 HP-UX 9.0 AIX 4.1 Figure 6: Throughput with 10 Mbps Ethernet, using the FILESNDL script. Units are shown in Mbps; higher is better. Testing the Performance of Multiple TCP/IP Stacks 7

8 NT WFW 3.11 OS/2 Warp 4.0 NetWare 4.11 Figure 7: Throughput with 100 Mbps Ethernet, using the FILESNDL script. Units are shown in Mbps; higher is better. Each test sent 1,460,000 bytes 100 times and timed each 1,460,000-byte transfer. Every stack (but OS/2) was able to use almost all the 10 Mbps link, with the three UNIX computers a little better than the rest. When we look at the 100 Mbps results, we begin to notice some differentiation among stack performance. Also the throughput performance does not scale directly when moving from 10 Mbps to 100 Mbps. It is much harder for a stack to take advantage of the 100 Mbps available. There are two surprising results in the 100 Mbps tests. First, the OS/2 throughput is poorer than expected. As a mature multitasking PC operating system, we expected its performance to be on par with NT and better than 95 and 3.1. When the same test is run with a shorter file size (100,000 bytes), the OS/2 throughput drops to 17 Mbps, while the other stacks only dropped by 5 or 6 Mbps (shown in Figure 8). The second surprise was the good performance seen by Microsoft s 32-bit TCP/IP stack for for Workgroups (WFW) With other 16-bit stacks on 3.1 or 3.11 using 100Mbps Ethernet, we saw little over 10 Mbps of throughput. With most 3.1 TCP stacks, upgrading to 100 Mbps was not advantageous; this was not the case with Microsoft s 32-bit stack. Tuning for Throughput There are four parameters that have a significant impact on the throughput of a TCP/IP stack. These parameters are file size, send buffer size, MTU, and window size. File size is how much user data to send from one program to another. The file size needs to be big enough to allow an accurate measurement of the network. Too small a size yields an unrealistically low measure of your network s throughput. We noticed a measurable increase in performance when file size is increased from 100,000 bytes to 1,460,000 bytes. The increase in performance for file sizes over 1,460,000 is negligible. 8 Copyright NetIQ Coporation,

9 k send size 100K send size NT WFW 3.11 OS/2 Warp 4.0 NetWare 4.11 Figure 8: Throughput with 100 Mbps Ethernet, comparing FILESNDL file sizes of 1,460,000 bytes and 100,000 bytes. Units are shown in Mbps; higher is better. Send buffer size is the number of bytes of user data provided on each TCP Send call. This number should be as large as possible; the fewer API crossings the better. On many operating systems, a Send call involves crossing from user space to kernel space. Once the data makes this crossing, it can be sent from the kernel very efficiently. We want to get as much data as possible to the TCP stack as we can. We used a 32 KB send buffer size in our tests. Many stacks allow a 64-KB send buffer size. MTU (maximum transmission unit) is the frame size allowed on the link. For Ethernet, this is normally already set to 1,500 bytes, which means 1,500 bytes is all that can be sent at a time. One 32- KB send will result in 25 MTUs. Notice that 32 KB divided by 1,500 is so how do we get 25 MTUs? 32 KB is the amount of user data requested to be sent. The 1,500-byte MTU size is the total data that can be sent on the link, which includes user data and protocol headers (20 bytes for TCP and 20 bytes for IP). The amount of user data that can be sent is thus 1,460 bytes. To improve throughput, the TCP stack should send full frames. To do this best, we send in multiples of 1,460. We used a file size of 1,460,000 with FILESNDL, so each frame sent would be full. To demonstrate the effect of not filling frames, we ran two tests, the first with a send buffer size of 1,460 (a full frame) and the second with a send buffer size of 1,461 bytes. Between our two NT computers, performance drops by over one Mbps on a 10 Mbps Ethernet link. To increase performance it is important to understand what is taking place on the underlying network, and then set your parameters accordingly. Testing the Performance of Multiple TCP/IP Stacks 9

10 bytes 1461 bytes Figure 9: Throughput on NT with 10 Mbps Ethernet, comparing send buffer sizes of 1,460 and 1,461 bytes (where the MTU was 1,500, including 40 bytes of header). Units are shown in Mbps; higher is better. Window Size is the amount of user data that can be sent before an acknowledgment is required. A common default for this parameter is 8,760 bytes, which means that after sending 8,760 bytes of data, the TCP stack must wait for an acknowledgment before proceeding. Acknowledgments are a safeguard against one computer sending data faster than the other computer can process it. To increase throughput, the number of acknowledgments should be low. If all computers can handle it, the window size can be increased close to 64 KB. With our computers, increasing the window size did not help performance noticeably after we had adjusted the other parameters. In a different configuration, such as multiple file transfers executing simultaneously, the window size was seen to have a greater effect. Response Time Our next set of tests was designed to show differences in response time among the stacks. Response time is the average time it takes to complete a transaction. We used the CREDITS script (short transactions which include the connection setup and takedown) in the same configuration as the throughput tests NT WFW 3.11 OS/2 Warp 4.0 NetWare 4.11 Solaris 2.4 HP-UX 9.0 AIX 4.1 Figure 10: Response time with 10 Mbps Ethernet, using the CREDITS script. Units are shown in seconds per transaction; lower is better. 10 Copyright NetIQ Coporation,

11 NT WFW 3.11 OS/2 Warp 4.0 NetWare 4.11 Figure 11: Response time with 100 Mbps Ethernet, using the CREDITS script. Units are shown in seconds per transaction; lower is better. Figure 11 shows that NT and OS/2 have a poorer response time than 95, 3.11, or NetWare To determine the cause, we closely compared NT and 95. First, we ran 10 concurrent pairs of the CREDITS script using NT and 95. We expected to see NT have a faster average response time than 95 because of NT s ability to handle multiple tasks better than 95. The results were the same as the single pair test. Next, we analyzed a line trace of NT and 95, using the CREDITS script. The line trace showed that the connection-setup processing for 95 is consistently faster than the NT connectionsetup processing. 95 took about 1 millisecond for connection setup, while NT took between 2 and 3 milliseconds. To validate this observation, we ran the CREDITL script between the five PC operating systems. The CREDITL script establishes a connection once and then sends and receives in the same manner as CREDITS. Running the test with CREDITL compares the data transfer without the connection overhead. As you can see in Figure 12, CREDITL improves on NT, but not on OS/2. We see that connection setup is slower on NT than on the other PC operating systems NT WFW 3.11 OS/2 Warp 4.0 NetWare 4.11 Figure 12: Response time with 100 Mbps Ethernet, using the CREDITL script. Units are shown in seconds per transaction; lower is better. Testing the Performance of Multiple TCP/IP Stacks 11

12 Performance Implications of the TCP Close We ve looked at performance influences of operating systems, protocol stacks, their tuning parameters, and the effects of application size parameters on data transfers. A surprising influence is the type of Close used for TCP connections. When short-connection TCP transactions are quickly repeated back-to-back, their transaction rate can significantly decrease over time depending on how the connections are closed. TCP offers two flavors of close: normal and abortive. Abortive Close gives the most consistent performance for repeated short connections. To illustrate the differences between these, we shall discuss the network flows and internal operation of the TCP stacks. TCP s Normal Close Operation A Normal Close consists of a three-way TCP close process, shown in Figure 13. One program (here located at side A) initiates the close at the application level by issuing a Close on the socket. This causes the FIN flag to flow to the TCP stack at side B. The FIN flag is acknowledged by the TCP stack at side B. The application at side B is only allowed to issue a Close, which causes a FIN flag to flow back to side A. While side B is able to completely close the connection, side A is not. The connection is left in TIME_WAIT state in case the ACK is not received by side B and side B retransmits the FIN. side A side B sockets internal TCP flag internal sockets call TCP state TCP state call close() FIN_WAIT_ FIN ---> CLOSE_WAIT <--- ACK ---- FIN_WAIT_2 close() LAST_ACK <--- FIN ---- TIME_WAIT ---- ACK ---> CLOSED Figure 13: The sockets calls, internal TCP states, and network flows for a Normal Close. While running tests with the CREDITS script between side A and side B, we saw the performance degrade as the test ran until it appeared to stabilize. Figure 14 was generated using the CREDITS scripts between two OS/2 computers. We saw similar results with other TCP stacks, although not as dramatic as exhibited by OS/2. Figure 14: Using TCP s Normal Close, the number of CREDITS transactions per second decreased as more were run back-to-back. The connection on side A is left in TCP s internal TIME_WAIT state, while the connection on side B is in CLOSED state. The connection on side A stays in TIME_WAIT state for 2 times the maximum segment lifetime (MSL). MSL is the maximum time a segment will live in the network. The reason for this is to be able to discard any spare segments that might exist in the network and to ensure side B receives the ACK. There has to be allowance for side B not receiving the ACK and thus retransmitting the FIN. 12 Copyright NetIQ Coporation,

13 As the number of connections in TIME_WAIT state increase, so does the processing overhead for each additional connection. Eventually connections are closing at the rate new connections are being started, and the performance levels off. TCP s Abortive Close Operation To achieve the best performance when running applications with short connections, it would be advantageous to close without leaving the connection in TIME_WAIT state. This is possible by using an Abortive Close, which causes an RST flag (reset) to flow across the network instead of a FIN flag. The RST flag immediately cleans up the connection. An Abortive Close is caused by setting the socket option SO_LINGER to 0 and then calling close. This solution is only appropriate in situations where the two partners in a connection know that all data has been sent and received. When the partners know for sure no more data will be sent on the session, they can get rid of the connection. In the defined scripts we used, we know that when the disconnect verb of the script is encountered, all data transfer verbs have already been issued. We just need a mechanism for ensuring that the data has arrived and been processed. Otherwise, an Abortive Close following a SEND can reset the connection before all the data is received by the partner. side A side B sockets internal TCP flag internal sockets call TCP state TCP state call recv() send() ===== the close processing starts here ======= recv() shutdown length=0 for write FIN_WAIT_1 <--- FIN ---- CLOSE_WAIT ---- ACK ---> setsockopt close() SO_LINGER=0 FIN_WAIT_2 close() CLOSED ---- RST ---> CLOSED Figure 15: The sockets calls, internal TCP states, and network flows for the Abortive Close. To ensure all data was sent and received, we implemented the following algorithm in the close processing shown above. If a RECEIVE was performed last, our program issues a recv to wait for the close to be issued by the other side. When the recv returns 0 bytes received, it issues an Abortive Close, which causes a RST flag to flow and clears the connection on the other side. If a SEND was performed last, our program issues a Shutdownfor-write, which forces the FIN flag to flow. Figure 15 illustrates this process. Figure 16 is the graph Chariot produced when close was implemented as seen in Figure 15. Figure 16: Using TCP s Abortive Close, the rate of CREDITS transactions remained constant as more were run. Conclusions We have shown that TCP provides a high level of performance and should be the protocol of choice over UDP for most applications. TCP performance is acceptable whenever the time for connection setup and takedown is trivial, such as in file transfers. If an application needs to avoid connection overhead, UDP can provide a performance gain. UDP can also be valuable for a specialized application that needs a reliable transport algorithm tuned for a specific type of data. Otherwise, the performance of TCP and UDP is almost identical when connection setup/takedown is factored out, regardless of LAN speed. We see that TCP provides very good performance and is easier for an application to use because of the built-in reliable transport. Testing the Performance of Multiple TCP/IP Stacks 13

14 When using 10 Mbps Ethernet, almost any operating system s TCP/IP stack is capable of using all the bandwidth with just one connection. When 100 Mbps Ethernet links are used, the performance difference between stacks can be easily seen and the choice of operating system becomes more important. This difference may become even more pronounced as faster networks become available. We have two caveats regarding the above observations. First, as we tested internally with faster PCs, we saw a significant improvement in performance on 100 Mbps Ethernet. With NT 4.0 on Pentium Pro 200 computers, we were able to attain over 80 Mbps throughput, compared to the 60 Mbps we saw with 166 MHz Pentiums. We have additional exploration to do in this area. Second, NetWare 4.11, which did the best at 100 Mbps in our tests, had a much higher default TCP Receive Window Size, which we did not reduce. We need to explore further the effect of this tuning parameter. We also looked at response time variation among operating systems. When connection setup/takedown was considered, NT and OS/2 had the slowest times. When connection setup/takedown was ignored, 95 and OS/2 were the slowest. Finally, when writing programs that do short, repeated connections over TCP, we recommend using the Abortive Close algorithm that we presented for closing Sockets. The Abortive Close algorithm can significantly improve the performance consistency of short connections when using TCP. We feel we have just scratched the surface in gaining an understanding of how these protocols behave on real computers and operating systems. All of the test results shown here involved just a single connection. As the number of concurrent connections is increased, we expect to learn more about the aggregate throughput and response times of these protocols. We also look forward to many lessons when intermediate devices, such as routers and switches, are included in the measurements. Appendix A: Test Methodology We ran our tests among a pool of computers on an isolated Ethernet network. Two identical computers were configured for testing the five PC operating systems: Dell Dimension XPS P166s, containing an Intel Pentium 166 MHz CPU, PCI bus, 32MB EDO RAM, and 3Com 3C905 Fast EtherLink 10/100 XL network adapter. The following five operating systems were loaded onto each of these two computers with the same hardware configuration: Microsoft NT Server 4.0 (with service pack 3) Microsoft 95 Microsoft for Workgroups 3.11 with Microsoft s TCP-32 stack (this stack was shipped with Microsoft s NT 4.0 Server) IBM OS/2 Warp 4 Novell NetWare 4.11 Three UNIX computers were used in the tests. Each had a 10 Mbps Ethernet adapter integrated with its motherboard. IBM RS/6000 POWER/PC, machine type 7248, with 64 MB RAM, running AIX version 4.1 HP model 712/60, with 32 MB RAM, running HP-UX version 9.0 Sun SPARCStation 5, with 64 MB RAM, running Solaris version 2.4 The tests run on the platforms, OS/2, and NetWare were run with identical computers as Endpoints 1 and 2. The tests performed with the AIX, HP-UX, and Sun Solaris, were run with NT as Endpoint 1 and the UNIX computer as Endpoint Copyright NetIQ Coporation,

15 Appendix B: Details of Test Scripts Here are the details of the three application scripts we used in our tests: CREDITS, CREDITL, and FILESNDL. In scripts identified as long connection, a single connection is used for the entire script, no Endpoint 1 Endpoint 2 number_of_timing_records=50 number_of_timing_records=50 START_TIMER transactions_per_record=25 transactions_per_record=25 CONNECT_INITIATE CONNECT_ACCEPT SEND RECEIVE size_of_record_to_send=100 size_of_record_to_send=100 send_buffer_size=default receive_buffer_size=default CONFIRM_REQUEST CONFIRM_ACKNOWLEDGE DISCONNECT DISCONNECT INCREMENT_TRANSACTION END_TIMER matter how many transactions are run. The time to start and stop the connections is not included in the timing results. For those identified as short connections, a new connection is started within each transaction. TCP and UDP have overhead associated with connection startup and takedown. Having these two variations of scripts lets you decide how much of the startup/takedown overhead to factor into performance measurements. CREDITS Script: This version of the Credit Check transaction uses short connections. This is a quick transaction that simulates a series of credit approvals. A 100-byte record is sent from Endpoint 1. Endpoint 2 receives the record and sends back a one-byte confirmation. Endpoint 1 Endpoint 2 CONNECT_INITIATE CONNECT_ACCEPT number_of_timing_records=50 number_of_timing_records=50 START_TIMER transactions_per_record=25 transactions_per_record=25 SEND RECEIVE size_of_record_to_send=100 size_of_record_to_send=100 send_buffer_size=default receive_buffer_size=default CONFIRM_REQUEST CONFIRM_ACKNOWLEDGE INCREMENT_TRANSACTION END_TIMER DISCONNECT DISCONNECT CREDITL Script: This version of the Credit Check transaction uses a long connection. This is a quick transaction that simulates a series of credit approvals. A 100-byte record is sent from Endpoint 1. Endpoint 2 receives the record and sends back a 1-byte confirmation. Testing the Performance of Multiple TCP/IP Stacks 15

16 Endpoint 1 Endpoint 2 CONNECT_INITIATE CONNECT_ACCEPT number_of_timing_records=100 number_of_timing_records=100 START_TIMER transactions_per_record=1 transactions_per_record=1 SEND RECEIVE file_size= file_size= send_buffer_size=default receive_buffer_size=default CONFIRM_REQUEST CONFIRM_ACKNOWLEDGE INCREMENT_TRANSACTION END_TIMER DISCONNECT DISCONNECT FILESNDL Script: This transaction uses a long connection to simulate sending a file from Endpoint 1 to Endpoint 2 and getting a confirmation back. We used file_size values of both 100,000 and 1,460,000 bytes in the tests described here. Operating System TCP Receive Window Size TCP send buffer size TCP receive buffer size UDP send buffer size UDP receive buffer size NT 4.0 8,760 32,767 32,767 8,183 8, ,760 4,096 32,767 8,183 8,183 WFW ,760 4,096 32,767 8,183 8,183 OS/2 Warp 4 28,672 32,767 32,767 8,183 8,183 NetWare ,768 32,767 32,767 8,183 8,183 AIX 4.1 8,760 32,767 32,767 8,183 8,183 HP-UX 9.0 8,760 32,767 32,767 8,183 8,183 Sun Solaris 2.4 8,760 32,767 32,767 8,183 8,183 These are the values we used for TCP Receive Window Size and the DEFAULT buffer sizes for the SEND and RECEIVE commands in the scripts (in bytes). Acknowledgments Luke S. Zettlemoyer happily handled the large task of designing and running the tests, and coordinating all the data. We could not have completed this work without him. We also appreciate the excellent feedback we received from our reviewers: Peter J. Schwaller, Joel Solkoff, Anne Schick, and Carl Lewis. An earlier version of this paper was presented at the Computer Measurement Group in 1997: John L. Wood, Christopher D. Selvaggi, and John Q. Walker II. Testing the Performance of Multiple TCP/IP Stacks, Proceedings of CMG97, December 7-12, 1997, volume 1, pages Bibliography 1) Comer, D. E. and D. L. Stevens. Internetworking with TCP/IP, Volume III. Prentice Hall, Englewood Cliffs, NJ, 1993 (ISBN ). 2) Quinn, B. and D. Shute. Sockets Network Programming. Addison-Wesley, Reading, MA, 1995 (ISBN ). 3) Stevens, W. R. TCP/IP Illustrated, Volume 1. Addison-Wesley, Reading MA, 1994 (ISBN ). 16 Copyright NetIQ Coporation,

17 Copyright Information NetIQ Corporation provides this document as is without warranty of any kind, either express or implied, including, but not limited to, the implied warranties of merchantability or fitness for a particular purpose. Some states do not allow disclaimers of express or implied warranties in certain transactions; therefore, this statement may not apply to you. This document and the software described in this document are furnished under a license agreement or a non-disclosure agreement and may be used only in accordance with the terms of the agreement. This document may not be lent, sold, or given away without the written permission of NetIQ Corporation. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, or otherwise, with the prior written consent of NetIQ Corporation. Companies, names, and data used in this document are fictitious unless otherwise noted. This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of the document. NetIQ Corporation may make improvements in and/or changes to the products described in this document at any time NetIQ Corporation, all rights reserved. U.S. Government Restricted Rights: Use, duplication, or disclosure by the Government is subject to the restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause of the DFARs and FAR (c) and any successor rules or regulations. AppManager, the AppManager logo, AppAnalyzer, Knowledge Scripts, Work Smarter, NetIQ Partner Network, the NetIQ Partner Network logo, Chariot, Pegasus, Qcheck, OnePoint, the OnePoint logo, OnePoint Directory Administrator, OnePoint Resource Administrator, OnePoint Exchange Administrator, OnePoint Domain Migration Administrator, OnePoint Operations Manager, OnePoint File Administrator, OnePoint Event Manager, Enterprise Administrator, Knowledge Pack, ActiveKnowledge, ActiveAgent, ActiveEngine, Mission Critical Software, the Mission Critical Software logo, Ganymede, Ganymede Software, the Ganymede logo, NetIQ, and the NetIQ logo are trademarks or registered trademarks of NetIQ Corporation or its subsidiaries in the United States and other jurisdictions. All other company and product names mentioned are used only for identification purposes and may be trademarks or registered trademarks of their respective companies. Testing the Performance of Multiple TCP/IP Stacks 17