Network Performance Evaluation of Latest dows Operating Systems Josip Balen, Goran Martinovic, Zeljko Hocenski Faculty of Electrical Engineering Josip Juraj Strossmayer University of Osijek Osijek, Croatia E-mail: josip.balen@etfos.hr, goran.martinovic@etfos.hr, zeljko.hocenski@etfos.hr Abstract: In this paper network performance evaluation of three latest versions of dows OSs; namely dows XP, dows Vista and dows 7 is conducted. Network performance measurements were obtained with D-ITG (Distributed Internet Traffic Generator) on a two identical computer systems connected with 1 Gbps Ethernet crossover cable. Following metrics were measured with OWD (One-Way- Delay) and RTT (Round-Trip-Time) meters: delay, jitter, throughput and CPU usage. With OWD meter performance of TCP protocol was measured when generating DNS-like traffic. Performance of UDP protocol was measured when generating two gamming traffic patterns and one VoIP call. With RTT meter performance of TCP and UDP protocols was measured for different packet sizes and exponential distribution of interdeparture time. Both TCP and UDP were running over the IPv4 and IPv6, respectively. The results show that network traffic with smaller packet sizes would benefit from dows Vista and dows 7. However, network traffic with bigger packet sizes still has the best performance in dows XP. 1. INTRODUCTION Since today the great majority of a personal computer systems are connected in some type of network (LAN, WLAN, Internet, etc.), there are more and more network applications and network performance is becoming very important. Mostly, network structure is very complicated and is composed of many different devices (routers, hubs, switches, servers, and computers). There are many various factors that affect the network performance, such as a network load and congestion, the number of users on the network, signal strength (for WLAN), etc. Even though network performance is mostly depending on a network hardware and equipment, OSs (Operating Systems) could have great impact as well. Furthermore, for the end users is very important that their computer systems are not the bottleneck in the network. Therefore, it is very important to properly select network equipment and an OS. Since dows OSs are the most widely used OSs for the personal computers, three latest versions of dows OSs are used in our experiments. Furthermore, since more and more dows OSs users are migrating from dows XP and dows Vista to dows 7, the goal of this study is to discover whether newer version of dows OSs provide better network performance. According to the Net Application statistics [1] in 211, dows XP had decreased number of users for 12.47%, dows Vista for 3.9%, meanwhile dows 7 increased number of users for 11.72% when looking at the total number of personal computer systems users. This is probably result of hardware improvements and frequently mentioned dows 7 performance improvements. Therefore, the main goal in this paper is to measure and evaluate the network performance of the three latest 32-bit versions of the dows OS; namely dows XP Professional SP3 (Service Pack 3), dows Vista Business SP2 and dows 7 Professional SP1. Performance measurement is provided with a D-ITG (Distributed Internet Traffic Generator) benchmark application in same and controlled environment for all three dows OSs. Most emphasis is put on a network metrics like delay, jitter and throughput. Furthermore, CPU usage is also measured in order to get insight which dows OS use more CPU resources for the network operations. With using more CPU resources for network traffic, lower amount is available for other tasks, and therefore overall computer system performance is lower. This paper continues our work in the area of network performance started in [2] where we preformed performance evaluation of the three latest versions of dows OSs in a five different evaluation areas: CPU scheduling, memory, graphics subsystem and hard disk drive management and network performance. In this paper we are using newer versions of dows updates, hotfixes and SPs in all three dows OSs. Furthermore, we are using different network performance measurement settings, and we are measuring network performance for more different traffic types. The rest of the paper is organized as follows. Section 2 brings related work. Performance measurement setup and methodology is detailed in Section 3. Section 4 presents performance measurement results and analysis. Section 5 concludes the study. 2. RELATED WORK Most significant papers dealing with the network performance of different OSs are presented below. IPv4 and S11-34685 - 139 SoftCOM 212
IPv6 performance evaluation on dows Vista and Linux Ubuntu for TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) traffic has been performed in [3]. Compared to Linux Ubuntu, dows Vista showed lower throughput and delay, and higher CPU usage. In [4], network performance evaluation of five different versions of dows OS was conducted. In most measurements dows Server 23 showed worse performance when compared to others. For TCP traffic the average difference between versions range from 2% to 5%, whereas for UDP traffic is around 3%. However, authors concluded that there is no clear winner. In [5], network performance evaluation of dows 23, dows XP and dows Vista in wireless LAN IEEE 82.11g environment is performed. The results indicate that dows Vista has lower performance in terms of the bandwidth and the round trip time than the other two dows OSs. Packet-forwarding performance evaluation of the Linux, dows Server and dows XP is preformed in [6]. In the case of kernel (or IP) packet forwarding performance measurement results indicate that Linux has the best packet-forwarding performance in terms of throughput, packet loss and delay. In the case of user application packet forwarding, dows Server showed smallest delays and highest throughput. In [7], the impact of the running CPUbound applications on the performance of IP packet forwarding (kernel level forwarding) in Linux and dows XP is studied. Performance measurement results show that Linux IP forwarding is not affected by running CPU-bounded applications, whereas dows XP network performance is degraded in terms of throughput, packet loss and delay. An evaluation and comparison study of Snort NIDS (Network Intrusion Detection System) performance under Linux and dows Server is performed in [8]. From the performance measurement results it can be concluded that Linux obtains better performance gain for Snort under malicious traffic while dows Server obtains better performance for Snort under normal traffic. 3. PERFORMANCE MEASUREMENT SETUP AND METHODOLOGY Measuring network performance with additional network equipment (switches, hubs, etc.) could have negative impact on the performance. Furthermore, to avoid problems with hardware incompatibility or lowering performance when using different hardware that might not function very well with each other, performance measurements are performed with the two identical computer systems directly connected with 1 Gbps Ethernet crossover cable (interfaces on both sides of the cable). They are equipped with Intel dual-core CPU (2.5 GHz), 4 GB of DDR2 RAM memory, embedded LAN Marvell88E856 PCI-E Gigabit LAN controller. All three evaluated OSs (dows XP Professional SP3, dows Vista Business SP2 and dows 7 Professional SP1) were updated with the latest updates in January 212 and 32-bit versions were used. D-ITG 2.6.1d described in [9], is used as primary network performance measurement tool since it works on dows OSs and is capable to produce different types of traffic and to measure network performance related metrics with two types of meter: OWD (One-Way-Delay) and RTT (Round-Trip- Time). By using two separate components called ITG-Send and ITG-Receive it can measure delay, packet loss, jitter and throughput. Furthermore, it supports various protocols such as TCP, UDP, ICMP (Internet Control Message Protocol), DNS (Domain Name System), Telnet and VoIP (Voice over Internet Protocol) with both IPv4 and IPv6 traffic generation. Traffic can be produced with various probability distributions: constant, uniform, exponential, Pareto, Cauchy, normal, Poisson and gamma. Performance measurements are performed with two different meters: 1. OWD meter: Performance of TCP protocol was measured when generating DNS-like traffic. Performance of UDP protocol was measured when generating two gamming traffic patterns (CounterStrike active and Quake 3) and one VoIP call (G.729 voice codec, 3 samples, 33 packets/s, framesize 1.). Both TCP and UDP were running over the IPv4 and IPv6, respectively. Before measuring network performance with OWD meter the real time clocks of the sender and receiver were synchronized. 2. RTT meter: Performance of TCP and UDP protocols was measured when running over the both IPv4 and IPv6, respectively. All measurements were conducted for 8 different packet sizes in the range from 64 bytes to 1536 bytes and packet size was constant during each measurement. Inter-departure time option was exponentially distributed and the average number of packets was 3 packets/s. Duration of each measurement was 3s and all other parameters in D-ITG were default. ITGSend component was used for sending packets from the first computer system to the ITGRecv component on the second computer system. ITGLog on sender and receiver computer systems was used for generating log file with the measurement results. In D- ITG three network performance metrics were used: (i) network delay; (ii) network jitter; and (iii) network throughput. Beside the performance indicators obtained from D-ITG, during the network performance measurement with RTT meter, CPU usage was also measured using the dows performance monitor. This measure indicates processor activity and displays the average percentage of busy time observed during the measurement interval. To ensure results stability, each measurement was repeated for five times in the same conditions. The final result of each performance measurement is expressed as an arithmetic mean S11-34685 - 139 SoftCOM 212
of five repetitions. Performance measurement results are reported with three significant digits since the fourth digit represents the difference in the results less than 1%, and consequently it enters the area of the measurement error. In Tables 1 and 2, which are reporting performance measurement results of all three dows OSs, the percentage error formula is used for calculation of the percentage difference between the measured values, as shown in (1). dows XP values are used as reference values, and dows Vista and dows 7 values are compared with respect to the dows XP values. Performance metrics Table 1 - OWD meter results for IPv4 XP Vista 7 Unit (Vista (7 DNS (TCP) Dealy 8.54 7.6 11.5 ms -17.33% 34.66% Jitter.45.84.72 ms 86.67% 6.% Throughput.94.99.97 Kbps 5.32% 3.19% Counter Strike (UDP) Dealy 1.1 5.84 13.1 ms -42.18% 29.7% Jitter.31.41.26 ms 32.4% -16.4% Throughput 9.89 5.57 11.3 Kbps -43.68% 14.26% Quake (UDP) Dealy 1. 7.12 15.4 ms -28.8% 54.% Jitter.2.56.58 ms 18.% 19.% Throughput 74.1 74. 74.2 Kbps -.13%.13% Voice (UDP) Dealy 1.6 1.6 18.5 ms.% 74.53% Jitter.65.6.3 ms -7.69% -53.85% Throughput 11.1 11.1 11.1 Kbps.%.% % 1% 1 Performance measurement process developed in [2] was applied. Since antivirus programs have influence on network performance they were not installed. Furthermore, beside measurement tool and Java there was no other application installed in the system. Java was installed since D-ITG GUI (Graphical User Interface) is written in Java. During performance measurement there was no user activity in the system. 4. RESULTS AND ANALYSIS Network performance measurement results for dows XP, dows Vista and dows 7 are presented below. For evaluation of dows OSs network performance, four main metrics are used: Delay - is calculated as the average of differences between receiving and sending times of packets. Unit is ms (millisecond) and lower delay means better performance. Jitter is the packet delay variation. Unit is ms and lower jitter means better performance. Throughput represents the average bitrate and is measured in Kbps (kilobit per second). Higher throughput means better performance. CPU usage - is the primary indicator of processor activity, and is measured as the average percentage of busy time observed during the network performance measurement with RTT meter. Lower CPU usage means that more CPU resources are available to other tasks in a computer system. Performance measurements with OWD meter for IPv4 under four different traffic types are shown in Table 1. It is worth nothing that in all four types of network traffic dows Vista shows lower delays when compared to dows XP and especially much lower when compared to dows 7. When generating two gamming traffic patterns dows 7 has more than double delay when compared to dows Vista. However, dows Vista shows highest values of jitter for DNS and CounterStrike traffic. Furthermore, dows Vista and dows 7 have almost three times higher jitter than dows XP for Quake traffic. Throughput values are similar for all three OSs expect when generating CounterStrike traffic where dows Vista shows 43.68% lower throughput and dows 7 14.26% higher throughput when compared to dows XP. Performance metrics Table 2 - OWD meter results for IPv6 XP Vista 7 Unit (Vista (7 DNS (TCP) Dealy 961 277 79.5 ms 71.18% 91.73% Jitter.43.46.59 ms 6.98% 37.21% Throughput 1.1 1.1.98 Kbps.% -2.97% Counter Strike (UDP) Dealy 96 256 76.7 ms 73.33% 92.1% - Jitter.63.34.58 ms 46.3% -7.94% Throughput 8.32 7.97 7.85 Kbps -4.21% -5.65% Quake (UDP) Dealy 959 256 78.7 ms 73.31% 91.79% Jitter.49.53.54 ms 8.16% 1.2% Throughput 74.4 74.1 74.4 Kbps -.4%.% Voice (UDP) Dealy 961 289 79.2 ms 69.93% 91.76% Jitter.55.58.62 ms 5.45% 12.73% Throughput 11.1 11.1 11.1 Kbps.%.% S11-34685 - 139 SoftCOM 212
Table 2 presents performance measurements with OWD meter for IPv6 under four different traffic types. When comparing delays between OSs, huge differences can be noticed. dows Vista shows almost four times and dows 7 even 12 times lower delays compared to dows XP. This huge difference indicates that dows Vista and dows 7 provide higher IPv6 QoS (Quality of Service) than dows XP. The main reason probably is that IPv6 is fully supported in dows XP only through SPs and is used in dual-stack architecture in contrast to dows Vista and dows 7 where IPv6 is integral part and is used in single unified dual-layer stack. Jitter values are slightly higher for dows Vista and dows XP except for the CounterStrike traffic where dows Vista and dows 7 show 46.3% and 7.94% lower jitter respectively. Throughput is similar in all OSs, however only for CounterStrike traffic dows XP shows approximately 5% higher throughput than dows Vista and dows 7. In TCP traffic performance measurements results shown in Figure 1, dows XP IPv6 shows lowest delays for the bigger packet sizes (from 768 to 1536 bytes). However, for the smaller packet sizes below 512 bytes dows XP has higher delays for IPv6. This is also proved in Table 2 where all generated traffic with IPv6 is consisting of smaller packet sizes (below 256 bytes). IPv4 TCP delays are again smaller in dows XP for bigger packet sizes over 124 bytes. Jitter values for TCP protocol show generally lower jitter in dows Vista and dows 7. Only for packet size of 128 bytes and IPv4 protocol jitter is higher in dows Vista and dows 7 for 12.5% and 14.17%, respectively. For the smaller packet sizes (below 768 bytes) the throughput values are similar in all dows OSs. 7 TCP - Delay.1 TCP - Jitter Delay (ms) 6 5 4 3 2 1 Jitter (ms).8.6.4.2. 25 TCP - Throughput 7 TCP - CPU Usage of Sender Throughput (Mbps) 2 15 1 5 6 5 4 3 2 1 6 5 4 3 2 1 TCP - CPU Usage of Receiver Figure 1. RTT meter performance measurment results for TCP S11-34685 - 139 SoftCOM 212
3. UDP - Delay.14 UDP - Jitter Delay (ms) 2.5 2. 1.5 1..5 Jitter (ms).12.1.8.6.4.2.. Throughput (Mbps) 25 2 15 1 5 UDP - Throughput 8 7 6 5 4 3 2 1 UDP - CPU Usage of Sender 6 UDP - CPU Usage of Receiver 5 4 3 2 1 Figure 2. RTT meter performance measurment results for UDP Meanwhile, for the bigger packet sizes dows XP shows highest throughput for IPv4 and dows 7 for IPv6. On the sender computer system for sending TCP network traffic dows Vista uses most CPU resources especially for packets over 256 bytes. Generally, dows XP gives most CPU availability for other tasks except for highest packet size of 1538 bytes in IPv6 where dows 7 uses lowest amount of CPU resources. Similar as on sender, on the receiver computer system dows Vista uses the most CPU resources for network traffic, in average 24% more than dows XP and 15% more than dows 7. UDP traffic performance measurements results are shown in the Figure 2. For smaller (64 bytes) and medium sized (512 and 768 bytes) packets dows Vista and dows 7 show approximately 15% lower delays than dows XP. However, for the highest packet sizes (128 and 1536 bytes) dows XP has much lower delays (up to 6%) when compared to dows Vista and dows 7. Jitter results show similar values for the packet sizes up to 124 bytes in all dows OSs. However, for the bigger packet sizes over the 124 bytes, dows Vista and dows 7 show up to four times higher jitter when compared to dows XP. Throughput of the medium sized packets (512 and 768 bytes) is slightly higher (from 5 to 1%) in dows Vista and dows 7. However, throughput of biggest packets is much higher (from 15% to 35%) in dows XP. CPU usage of sender computer system is lowest in dows 7 for IPv6 and generally the highest in dows Vista for both IPv4 and IPv6. The lowest CPU usage on the receiver computer system for the packets up to 124 bytes for both IPv4 and IPv6 is obtained in dows XP. For the bigger packets over the 128 bytes the lowest CPU usage is accomplished with dows 7. S11-34685 - 139 SoftCOM 212
5. CONCLUSION This paper presents network performance evaluation of three latest versions of dows OSs; namely dows XP, dows Vista and dows 7. The network performance was primarily measured and evaluated with D-ITG in terms of network delay, jitter and throughput. Furthermore, we also measured the CPU usage to get insight into CPU availability during network testing. Experiment was conducted with OWD and RTT meter for TCP and UDP running over the IPv4 and IPv6, respectively. OWD results for IPv4 show that dows Vista has the lowest delays for all four types of generated traffic while dows 7 has the highest delays. For IPv6 dows Vista shows almost four times and dows 7 even 12 times lower delays compared to the dows XP delay values. However, jitter values are mostly lowest in dows XP for both IPv4 and IPv6. Therefore, we can conclude that newer versions of dows OSs (Vista and 7) with fully support for IPv6 have enhanced performance for IPv6 networking in terms of the network delays. TCP results indicate that dows XP has the lowest delays for bigger packet sizes for IPv4 and particularly for IPv6. However, this is not result with the jitter and throughput where dows XP with IPv6 has worse results than dows Vista and dows 7. For smaller packet sizes dows Vista and dows 7 obtain lower delays. Furthermore, CPU usage is highest in dows Vista and lowest in dows XP. Therefore, network-intensive applications that require low delays would benefit from dows XP IPv6 only if containing mostly bigger TCP packets. However, any other TCP based network applications would probably have similar performance in the all three dows OSs. For UDP network traffic with bigger packet sizes the best performance are obtained in dows XP. However, CPU usage for bigger packet sizes is also higher in dows XP than in dows Vista and dows 7. Therefore, it can be concluded that for the UDP network traffic with the bigger packet sizes more CPU resources are assigned to the network application in dows XP than in the other two dows OSs. Network applications that generate UDP traffic with smaller packet sizes will benefit from dows Vista and dows 7. REFERENCES [1] Net Applications: Top Operating System Share Trend ; January 212. available at http://www.netmarketshare.com /os-market-share.aspx?spider=1&qprid=11&qptimeframe=m &qpsp=144&qpnp=12 [2] Martinovic, G., Balen, J., Cukic, B.: Performance Evaluation of Recent dows Operating Systems, Journal of Universal Computer Science (JUCS), 18, 2 (212), p.p. 218-263. [3] Narayan, S., Shang, P., Fan, N: Performance Evaluation of IPv4 and IPv6 on dows Vista and Linux Ubuntu ; In Proceedings of the International Conference on Networks Security, Wireless Communications and Trusted Computing, Wuhan, (29), p.p. 653-656. [4] Narayan, S., Shi, Y.: TCP/UDP network performance analysis of windows operating systems with IPv4 and IPv6 ; In Proceedings of the 2nd International Conference on Signal Processing Systems, Dalian, (21), V2-219-V2-222. [5] Kolahi, S. S., Narayan, S., Nguyen, D. D. T., Sunarto, Y., Mani, P.: The Impact of Wireless LAN Security on Performance of Different dows Operating Systems ; In Proceedings of the IEEE Symposium on Computers and Communications, Marrakech (28), p.p. 26-264. [6] Salah, K., Hamawi, M.: Comparative packet-forwarding measurement of three popular operating systems ; Journal of Network and Computer Applications, 32, 5 (29), p.p. 139-148. [7] Salah, K., Hamawi, M.: Impact of CPU-bound Processes on IP Forwarding of Linux and dows XP ; Journal of Universal Computer Science, 16, 21 (21), p.p. 3299-3313. [8] Salah, K., Kahtani, A.: Performance evaluation comparison of Snort NIDS under Linux and dows Server ; Journal of Network and Computer Applications, 33, 1 (21), p.p. 6-15. [9] Botta, A., Dainotti, A., Pescape A.: Multi-protocol and multi-platform traffic generation and measurement ; IEEE Conference on Computer Communications, DEMO Session, Anchorage, (27). S11-34685 - 139 SoftCOM 212