WI-FI PERFORMANCE BENCHMARK TESTING: Aruba Networks AP-225 and Cisco Aironet 3702i

Transcription

1 WI-FI PERFORMANCE BENCHMARK TESTING: Networks AP-225 and Cisco Aironet 3702i Conducted at the Proof-of-Concept Lab January 24, 2014

2 Statement of Test Result Confidence makes every attempt to optimize all vendors for performance and follow best practices for configuration as published by the vendor. ensures that tests are performed in a fair manner including AP mounting position, client location, transmit power, channel and the latest firmware available from each vendors support website at the time of testing. ensures the test bed environment to be free of any interference sources. Also, the Wi-Fi channels configured are ensured to be consistent when testing 2.4 GHz and 5 GHz bands for all vendors. believes the test results are both repeatable and reproducible in similar testing environments. Configurations used in the testing for both vendors are posted here: WiFi_Performance_APPENDIX.pdf Table of Contents Executive Summary... 3 Test Environment... 4 Test Scenarios... 6 What Do The Tests Reveal? About Networks, Inc Networks, Inc. 2

3 Executive Summary The key enhancement that ac delivers in comparison to the n standard is the support of higher data rates. This is achieved mainly by doubling the maximum channel width to 80MHz as well as the addition of a more efficient modulation technique, 256-QAM, which is 33% more efficient. With a 3 spatial stream ac device it is now possible to achieve wireless data rates up to 1.3 Gbps on the 5 GHz radio. However, the data rates are only part of the picture. Although it is possible to achieve these maximum data rates with many of the ac access points currently in the market this does not mean they provide equal throughput. For the end user the high data rate has very little value if the actual throughput seen is still low. Real world traffic is composed of a mix of packets of various sizes so we have performed tests of several packet sizes here. Small packets are very demanding of the AP and require a powerful CPU in order to deliver the maximum performance. The purpose built AP-225 that was tested has been designed to provide not only high data rates but also high throughput with real world traffic with a mix of small, medium and large size packets. As part of the benchmarking process, the tests were run with both an Networks AP-225 and a Cisco Aironet 3702i access point. The key highlights from the testing are as follows. Table 1: Test results summary Test Scenario AP-225 Cisco 3702i Advantage Single Client downstream TCP throughput at 10ft % Single Client upstream TCP throughput at 30ft % Multi-client bidirectional TCP throughput with 30 clients using large packets (1500B) Multi-client downstream UDP throughput with 30 clients using small packets (256B) Multi-client downstream TCP throughput with 30 clients when co-channel interference is present % x x These test results indicate a significant variation in performance between and Cisco products. Customers are advised to exercise their own judgment before making a vendor decision for their Wi-Fi network. The rest of the document provides comprehensive details of the test cases, test bed setup, observations and results collected. Networks, Inc. 3

4 Test Environment Table 2: Devices under test Vendor Device Quantity Firmware version AP build Mobility Controller build Cisco Aironet 3702i WLAN Controller The latest firmware versions available through the support sites of each vendor were used. The following table shows the detailed information on various network components that were part of the infrastructure used for performance tests. Table 3: Test equipment Item Component Specifications Details 1 Device OS/Type Laptops Make and Model Quantity OS Version MacBook Pro (3SS ac) MacBook Air (2SS ac) Performance evaluation tools IxChariot, one wired endpoint, client on each device 7.10 SP5 3 Switch Cisco Catalyst 3750G 12.2 (50) SE5 4 Power Injector PowerDsine 9001G PoE+ injector N/A 5 Co-Channel Interference source AP-225, 7210 Mobility controller build Networks, Inc. 4

5 Test Bed Setup Figure 1 below shows the logical network topology of the test bed. The wireless controllers, IxChariot wired endpoint and access points were all connected at Layer 2 using a Cisco Catalyst 3750G switch. A PoE+ (802.3at compliant) power injector was used to power the access points. The access points were mounted on the ceiling and installed in the same location for the duration of that vendor test. Static IP addresses were used on the clients. Legend Figure 1: Test bed setup Networks, Inc. 5

6 Test Scenarios The following scenarios were benchmarked in the proof of concept lab. 1. Single Client TCP throughput at 10ft and 30ft 2. Multi-client TCP throughput with 30 clients using large packets 3. Multi-client UDP throughput with 30 clients using small packets 4. Multi-client (30) TCP throughput when co-channel interference is present During all tests, controllers were configured to use an 80MHz wide channel, and ac beamforming was enabled. Channel 100E was used in this test. The 2.4 GHz radio was also enabled but no clients were associated with the AP on 2.4 GHz band during this test. Networks, Inc. 6

7 Scenario 1: Single Client TCP throughput at 10ft and 30ft This test measures the maximum TCP throughput for a single client at two client locations; 10ft and 30ft away from the AP located in a typical office cube with line of sight (LOS). The client used in this test was a MacBook Pro supporting ac and three spatial streams capable of data rates up to 1300Mbps. TCP traffic was generated between the wireless and wired endpoint using Ixia IxChariot, High Throughput TCP script, 1MB file size, 20 pairs per client and run time of one minute. The test was repeated 3 times and the average of these three runs was recorded. The test results are shown in Table 4 below. Table 4: Single Client TCP Throughput at 10ft and 30ft Test Case AP-225 Cisco 3702i Advantage Downstream TCP Throughput 10ft LOS % Upstream TCP Throughput 10ft LOS % Bidirectional TCP Throughput 10ft LOS % Downstream TCP Throughput 30ft LOS % Upstream TCP Throughput 30ft LOS % Bidirectional TCP Throughput 30ft LOS % 1000 Single Client Throughput (Mbps) AP Cisco 3702i ' LOS 10' LOS 10' LOS 30' LOS 30' LOS 30' LOS Networks, Inc. 7

8 As can be seen from the results above, the AP-225 performance is superior to the Cisco 3702i in all test cases. Both APs achieved the maximum 1300Mbps data rates but the AP-225 provided up to 57% higher throughput compared to Cisco 3702i. The maximum TCP downstream throughput with the AP-225 was 882Mbps, which is approaching the limits of wired Gigabit Ethernet. Maximum downstream TCP throughput of the Cisco 3702i was a modest 574Mbps, 308Mbps lower than throughput with AP-225. While the AP-225 showed up to 42% higher throughput at 30ft compared to the Cisco 3702i in real world deployments this difference is most likely bigger. ClientMatch technology helps to keep the clients connected to the most optimum AP. Because of this in real world deployments you seldom see clients connecting to APs that are further than 30ft away from the AP. Cisco does not currently have ClientMatch equivalent functionality and sticky clients can sometimes be associated with AP quite far from the client degrading the performance seen by these clients and the whole network. Networks, Inc. 8

9 Scenario 2: Multi-client TCP throughput with 30 clients using large packets The second scenario evaluates the performance with multiple clients. A total of 30 clients were associated with the AP, up to 15 ft from the AP and had a clear line of sight to the AP. These 30 clients consisted of a mix of 10 MacBook Pro laptops (802.11ac, 3 spatial streams) and 20 MacBook Air laptops (802.11ac, 2 spatial streams). TCP traffic was generated between the wireless and wired endpoints using Ixia IxChariot, High Throughput TCP script, 1MB file size, 4 pairs per client and run time of one minute. The test was repeated 3 times and average of these three runs was recorded. The test results are shown in Table 5 below. Table 5: Multi-client TCP throughput with 30 clients using large packets Test Case AP-225 Cisco AP3702i Advantage Downstream TCP Throughput (Mbps) % Upstream TCP Throughput (Mbps) % Bidirectional TCP Throughput (Mbps) % 500 Multi-client TCP Throughput (Mbps) Downstream Upstream Bidirectional AP Downstream Upstream Bidirectional Cisco 3702i Networks, Inc. 9

10 Scenario 3: Multi-client UDP throughput with 30 clients using small packets This scenario evaluates the AP s ability to handle small packets. A total of 30 clients were associated with the AP all within 15 ft of the AP and with clear line of sight. The 30 clients consisted of a mix of 10 MacBook Pro laptops (802.11ac, 3 spatial streams) and 20 MacBook Air laptops (802.11ac, 2 spatial streams). UDP traffic was generated between the wireless and wired endpoints using Ixia IxChariot, modified UDP throughput script with either 256B packets (4 pairs per client, 2Mbps per pair) or 512B packets (4 pairs per client, 4Mbps per pair) and run time of one minute. The test was repeated 3 times and average of these three runs was recorded. The test results are shown in Table 6 below. Table 6: Multi-client UDP throughput with 30 clients using small packets Test Case AP-225 Cisco AP3702i Advantage Downstream UDP Throughput (Mbps) with 256B packets x Downstream UDP Throughput (Mbps) with 512B packets x 350 Multi-client UDP Throughput (Mbps) AP Cisco 3702i 512B Packets 157 AP B Packets As can be seen from the results, the AP-225 performance with small packets is significantly superior to the Cisco 3702i. What is especially notable there is the throughput with 512B packets. In today s enterprise networks the average packet size of typical traffic is closer to 512B than 1500B. While the AP-225 UDP throughput with 512B packets was not far Networks, Inc. 10

11 from the TCP throughput with large (~1500B) packets, the throughput of the Cisco 3702i totally collapses. As these are test cases that cause quite a high load on the CPU we wanted to also see the CPU utilization of the APs during these tests. These are shown in the figures below. Cisco CPU 90%+ utilization Figure 2: CPU utilization of Cisco 3702i during UDP throughput test with 256B packets CPU 75% utilization Figure 3: CPU utilization of AP-225 during UDP throughput test with 256B packets As can be seen from the figures above the CPU utilization of the Cisco 3702i is really high (>90%) and approaching 100% in both test cases. The high CPU utilization clearly shows that the CPU just cannot handle the load and that is leading to a significant performance degradation for the Cisco 3702i. The AP-225 can deliver high throughput in both test cases and the CPU peaks at a maximum utilization of 75% (>25% idle). This enabled the AP-225 to deliver high performance even with small packets while also leaving room for adding new features in the future without sacrificing performance. Networks, Inc. 11

12 Scenario 4: Multi-client (30) TCP throughput when co-channel interference is present This scenario focuses on testing the AP performance when co-channel interference is present. In real world environment this can happen e.g. when people bring in their own mobile hotspots like the MiFi, or enable personal hotspot feature on their smartphones. As in scenario 2, a total of 30 clients were associated with the AP all within 15ft from the AP and with clear line of sight. The 30 clients consisted of a mix of 10 MacBook Pro laptops (802.11ac, 3 spatial streams) and 20 MacBook Air laptops (802.11ac, 2 spatial streams). To generate co-channel interference (CCI) a separate AP-225 was used connected to an 7210 Mobility controller other than the device under test. The AP used as a CCI source was located ~15ft away from the AP under test, configured to use the same channel as the AP under test (100E). The signal strength observed at the location of the AP under test was around -60dbm. Two clients (802.11ac capable MacBook Pros) were then associated with the AP used as a CCI source and traffic was generated using IxChariot to cause ~20% channel utilization. TCP traffic was generated between the wireless and wired endpoints using Ixia IxChariot, High Throughput TCP Script, 1MB file size, 4 pairs per client and run time of one minute. The test was repeated 3 times and average of these three runs was recorded. The test results are shown in Table 7 below. Table 7: Multi-client TCP throughput with 30 clients when co-channel interference is present Test Case AP-225 Cisco AP3702i Advantage Downstream TCP Throughput (Mbps) % Upstream TCP Throughput (Mbps) % Bidirectional TCP Throughput (Mbps) % The impact of the co-channel interference on the throughput is shown in the Figure 4 below when comparing these results with the test run when interference was not present. Networks, Inc. 12

13 450 Downstream Throughput (Mbps) % % AP-225 Cisco 3702i AP-225 Cisco 3702i Without Interference With Interference Figure 4: Impact of co-channel interference on throughput Once again the AP-225 clearly outperforms the Cisco 3702i. The Cisco 3702i throughput drops 41.7% in the presence of co-channel interference whereas the AP-225 throughput dropped only 7.6%. Even in the presence of co-channel interference the AP- 225 provides higher throughput than the Cisco 3702i in clean spectrum. Networks, Inc. 13

14 What Do The Tests Reveal? The tests conclusively show that the AP-225 is superior in performance to the Cisco 3702i in all test scenarios. Many of these test cases cause considerable load on the CPU of the access point. This is also one of the areas were we see biggest differences in design of these two access points. Whereas the AP-225 has been designed from ground up to meet the demands of ac networks using the latest processor technology, Cisco has opted to use the same old CPU on the 3702i that they used two years ago on the 3602i. As a result the AP-225 has double the CPU capacity of the Cisco 3702i. This higher CPU capacity is required on ac access points to deliver what the ac standard promises. This is evident in the test results. The maximum TCP downstream throughput of Cisco 3702i was capped at modest 574Mbps. However, the AP-225 was able to deliver an impressive 882Mbps of TCP throughput, which is something one expects with physical layer data rate of 1300Mbps. The throughput test with small packets really highlights the differences in performance of these access points. With 256B packets, the AP-225 delivered 3.6 times the throughput of the Cisco 3702i. With 512B packets, the AP-225 throughput was 2.7 times the throughput of the Cisco 3702i. The average packet size in real world enterprise networks is still much closer to 512B than it is to 1500B, so this performance difference with 512B packets can be considered to reflect the performance difference of these two access points in real world deployments. Co-channel interference is quite often unavoidable, and commonly seen in the enterprise networks. The test results show that the AP-225 maintains excellent performance even in the presence of co-channel interference, and delivered throughput that exceeded the throughput that the Cisco 3702i had in clean spectrum environment. In the presence of cochannel interference the Cisco 3702i throughput dropped by 41.7% to only 187Mbps with 30 clients. This level of performance is achievable with older 3-stream n access points in a clean spectrum environment, which is not necessarily what you want to see after investing in new ac access points. Networks, Inc. 14

15 About Networks, Inc. Networks is a leading provider of next-generation network access solutions for the mobile enterprise. The company s Mobile Virtual Enterprise (MOVE) architecture unifies wired and wireless network infrastructures into one seamless access solution for corporate headquarters, mobile business professionals, remote workers and guests. This unified approach to access networks enables IT organizations and users to securely address the Bring Your Own Device (BYOD) phenomenon, dramatically improving productivity and lowering capital and operational costs. Listed on the NASDAQ and Russell 2000 Index, is based in Sunnyvale, California, and has operations throughout the Americas, Europe, Middle East, Africa and Asia Pacific regions. To learn more, visit at For real-time news updates follow on Twitter and Facebook Networks, Inc. Networks trademarks include AirWave, Networks, Wireless Networks, the registered the Mobile Edge Company logo, Mobility Management System, Mobile Edge Architecture, People Move. Networks Must Follow, RFProtect, and Green Island. All rights reserved. All other trademarks are the property of their respective owners. Networks, Inc. 15