High Density WLAN Comparison Testing: Aruba, Cisco and Juniper. September Copyright 2013 Novarum Inc.

Transcription

1 High Density WLAN Comparison Testing: Aruba, Cisco and Juniper September 2013 Copyright 2013 Novarum Inc.

2 Executive Summary 2 Key Findings 2 Test Environment 3 The Facility 3 Infrastructure Equipment 5 Client Equipment 5 Test Network 6 Testing Methodology 6 Tools 6 WLAN Infrastructure Configuration 7 Client Configuration 9 Comparison of High Density Testing 10 Comparison of Aggregate Capacity 10 Comparison of Retry Rates 13 Comparison of Test Streams 15 A Partial Analysis of Cisco Results 22 Conclusions 23 Key Findings 23 Appendix A About Novarum 24 Appendix B Aruba Configuration 25 Appendix C Cisco Configuration 39 Appendix D Juniper Configuration 43 High Density Wireless LANs: A Comparison 1

3 Executive Summary There has been an explosion of mobile devices that are used within enterprises and public spaces and almost all of these devices are wireless LAN-enabled. This concentration of wireless LAN devices imposes unique stresses on infrastructure wireless LANs, requiring them to operate at client and data densities that few networks ever have before. We chose to study this unique high-density environment by placing over 300 Wi-Fi client devices in a large room and driving varying traffic loads across subsets of this collection of devices. We evaluated three different wireless network infrastructures from leading enterprise network vendors - Aruba, Cisco and Juniper - with this collection of user devices and traffic loads. This paper describes the results of this comparative test of the three leading WLAN infrastructure systems in a high density environment. This is the second in a series of high density WLAN test reports conducted by Novarum. Please be sure to download first report: Novarum High Density WLAN Testing. Key Findings Our testing regimen clearly pushed the abilities of each of these networks and it was clear we had found the maximum capacities of each network. But the networks differed dramatically in where and how maximum capacity was handled. Each of these networks was clearly at maximum load, with network performance at peak as measured by the degradation of aggregate throughput and increased MAC frame retry rates as we increased the number of client devices. Juniper maintained consistent performance as load increased from under 100 clients to over 300 clients - delivering not only the highest absolute throughput, but with the best underlying fundamentals as evidenced by the lowest MAC frame retry rates and the lowest rate of stream failures as load increased. Its maximum performance was at over 50% more clients than Aruba s maximum performance and at client numbers and total traffic loads for which the Cisco network could not even be tested. For high density user device loads, Juniper clearly exhibited the most robust performance. Aruba delivered materially lower performance than Juniper - not only lower throughput but, more importantly, with fewer clients. We noted that the Aruba system had materially higher MAC frame retry rates (1.9x) and materially higher data stream failures (about 2x on average) than Juniper. The higher stream failure rates indicate that many more clients were starved for data throughput capacity under the Aruba system than the Juniper system. The Aruba system did successfully complete the full range of tests. While the Aruba system did not collapse like the Cisco system, it delivered lower maximum performance than the Juniper system and delivered that maximum performance with a lower number of client devices. The Cisco network was challenged by this test and did not come close to completing the full range of tests. We had great difficulty running the tests due to the very high number of stream failures that substantially increased the time necessary to run the test. With limited availability of the test facility, we were unable to complete a full suite of tests for the Cisco network. However, the one completed test is suggestive. Under a maximum client load of 302 clients, with the downstream-only traffic load, the Cisco network had over 62% stream failures, more than 62% of the test throughput streams had zero measurable data throughput. This compares to 17% stream failures for Aruba and 11% stream failures for Juniper. With almost two thirds of the test streams with zero throughput, the Cisco system can be considered to have collapsed under the network load. High Density Wireless LANs: A Comparison 2

4 Test Environment The Facility We conducted the testing in the Juniper Aspiration Dome, a conference facility in Sunnyvale, California, on the Juniper Networks campus adjacent to Moffett Federal Airfield. Figure 1 - The Aspiration Dome. The seating in the dome can be reconfigured to match the needs of specific events. For our testing the dome was configured as a large auditorium. There was a grid of seats, 20 per row, in the center of the Stage facility. We placed a single client device on each seat. The seats were numbered 1 through 20, and we used rows a through q. The client name was set to match its seat number: a1, a2, a3, etc. Every time we set AP1 AP4 up the tests, the clients were placed in the same location. AP2 Clients Clients AP5 AP3 AP6 Chariot and Monitor Machines Figure 2 - Client Devices in Rows Figure 3 - Dome Layout High Density Wireless LANs: A Comparison 3

5 During our testing, the permanent Juniper access points in and around the facility were disabled and we verified with our tools that there were no other Wi-Fi networks that would interfere with our testing. Figure 4 - Access Point Mounting The access points under test were mounted on temporary stands about 10 feet above floor level as shown in Figure 4. At the base of each stand we had a MacBook Pro laptop running a packet capture program that was monitoring the 5 GHz channel being used by that AP. High Density Wireless LANs: A Comparison 4

6 Infrastructure Equipment The Wi-Fi infrastructure for this testing is current three stream n enterprise Wi-Fi systems. The detailed configurations are contained in an appendix later in this document. Juniper WLC880 Wireless LAN Controller WLA532 Access Point Cisco WLAN Controller 5508 AP 3602 Aruba WLAN Controller 3600 AP 135 Client Equipment One of the significant factors in this high-density testing is our use of real-world client machines rather than traffic simulators. We used more than 300 Wi-Fi clients. There are three categories of clients - laptops, smartphones and tablets. We had a variety of clients running different operating systems: Windows 7 laptops from Dell and HP; MacBook Pros and MacBook Airs; ios, Windows 8 and Android tablets such as the ipad, ipad Mini, Kindle Fire, Galaxy Tab 10.1, Microsoft Surface and Surface Pro; and ios and Android smartphones including iphone 3, 4, and 5, Samsung Galaxy 3s, Nexus and more. These clients included all varieties of Wi-Fi devices that will typically be found in a modern BYOD environment: 2.4 GHz n 1x1 MIMO smartphone clients Dual band 2.4 and 5 GHz 1x1 MIMO 20 MHz tablets Dual band 2.4 GHz and 5 GHz 1x1 MIMO 40 MHz tablets Dual band 2.4 GHz and 5 GHz 2x2 MIMO laptops Dual band 2.4 GHz and 5 GHz 3x3 MIMO laptops High Density Wireless LANs: A Comparison 5

7 Test Network The test network configuration is shown in Figure 5. This is a high-availability, high-density configuration for the entire network. There are redundant routers, redundant Ethernet switches and redundant wireless LAN controllers. Figure 5 - Test Network Topology Testing Methodology Tools We used Ixia Chariot to generate traffic and record throughput. Chariot generates different network traffic flows between pairs of endpoints. Every client machine had the Ixia Endpoint software installed. We had a dedicated hardware Ixia Chariot server with licenses for up to 500 endpoints. We collected packet captures in the 5 GHz band from each AP on each 5 GHz channel. During each test run, we captured packets for 30 seconds in the middle of the data transfer. We extracted data and analyzed the packet captures using EyePA and Cascade Pilot. High Density Wireless LANs: A Comparison 6

8 We used Channelyzer Pro and AirMagnet from Fluke to do spectrum analysis. The spectrum screen shots in this paper are from Channelyzer Pro. We drove the network with traffic generated from three different Chariot scripts: Max Throughput Script The "Max Throughput Script" creates a single TCP stream for smartphones and tablets. Each laptop client was configured to support two TCP streams. All TCP streams are running the High Throughput TCP script from the Chariot server to the client. The high throughput script attempts to send as much data as possible. Bi-Directional Script The "Bi-Directional Script" is a single TCP stream per client for smartphone and tablets, with each one running High Throughput TCP script from the Chariot server down to the client. For laptops there are two TCP streams per client, one running the High Throughput TCP script down from server to client and one running the script up from the client to server. The laptops are the only clients with bidirectional traffic in this test. Low Throughput Script The "Low Throughput Script" is a single TCP stream per client for all clients. All TCP streams are running the High Throughput TCP script from the Chariot server to the client; however, the throughput per client is constrained. For laptops, the throughput is constrained to 5 Mbps per stream, tablets are limited to 2 Mbps and smartphones are limited to 1 Mbps. The scripts are configured to run for two minutes, with each client generating continuous streams of traffic. This is one area where this high-density testing goes beyond the typical enterprise usage scenarios. Most wireless LANs experience bursty traffic - some very high throughput peaks, followed by longer periods of low activity. Constant load from all clients for a two minute period is very unusual, and difficult for networks to handle. With bursty traffic, the pressure on the network is relieved during the gaps in offered load. With continuous offered load, retransmissions compete with new transmissions, congestion builds and there is no opportunity to relieve this pressure. This extreme traffic pattern generated by our test scripts is useful for high-density testing because it allows us to simulate what would happen to these networks with many more clients running more typical network traffic patterns. Each time we conducted a test, we would run the test at least three times to ensure that we were getting consistent results. With over 300 clients of different types with different operating systems, perfection was not possible. We spent a great deal of time configuring the clients to ensure that they were active on the network and running the proper software. Before running each series of tests we ran a script that would contact each client to make sure that it was active, on the network and running the Chariot endpoint software. However, there were always a few clients that would not initialize properly and did not complete the tests. The clients could be unavailable for a variety of reasons: their battery died; they are off-line downloading new firmware; their Wi-Fi turned off; or their network protocol stack was not running correctly. We decided that this was a realistic part of the test and since the number of such off line clients was always small, it did not materially affect the results of the test. WLAN Infrastructure Configuration We followed the manufacturer recommendations and guidelines for high-density deployments. We configured the APs manually with static channel assignment and power levels even though the vendors normally recommend automatic channel tuning. We wanted to maintain a consistency of configuration that would be repeatable throughout the testing, and we did not want any channel changes to occur during the testing. High Density Wireless LANs: A Comparison 7

9 Our testing area inside the Dome was a large open space. Every client device was close enough to hear every AP, and the APs were all within range of each other. In order to provide maximum capacity, we wanted to avoid sharing the same channels between APs. This is not possible in the 2.4 GHz band. With only three independent Wi-Fi channels available, we had to use each 2.4 GHz channel twice. We separated the APs operating on the same channel as far as possible, but they were clearly overlapping and any associated clients will also share the same channel. This 2.4 GHz configuration is not ideal, but it exemplifies a real world constraint that most enterprise networks have to address. High-Density Channel Configuration At 5 GHz, there is more unlicensed spectrum and more independent channels are available. For this high-density environment we wanted as many independent channels as possible. Since our client devices included many modern n clients, we also wanted to use 40 MHz wide channels in the 5 GHz band for the highest data rates possible. We initially configured six independent 40 MHz channels in the 5 GHz band, one for each AP. The high-density channel layout for the APs is shown in the table at right. This channel configuration at 5 GHz used two Dynamic Frequency Selection (DFS) channels. In order to use these DFS channels, the system must detect radar operating in those frequencies. If radar is detected, the wireless LAN must abandon that channel and move to an alternate channel. During the network configuration and test, we experienced a few radar events a day. (The Dome is located very close to an active airport, Moffett Federal Airfield.) These DFS events modified the test results and made the testing less repeatable. Overall, using the DFS channels proved to be too disruptive and time consuming during our testing. Therefore we changed the 5 GHz channel plan to avoid the DFS channels and moved to a fixed configuration that was of lower capacity for most of the testing. Fixed 5 GHz Channel Configuration For this comparative multi-vendor testing we adopted a fixed 5 GHz channel configuration that avoids the DFS channels. This fixed configuration is 240 MHz total. The 2.4 GHz is the same - three 20 MHz channels shared by six APs. The 5 GHz band is four independent 40 MHz channels and one 20 MHz channel. Channel 165, the 20 MHz channel, is shared by two APs. The channel layout is shown in the table at right. Overall, the potential capacity for the system under test was reduced by 20% to 25% compared to the highest capacity configuration using 180 MHz of 5 GHz spectrum rather than potentially 240 MHz of spectrum. That is acceptable for this high-density WLAN testing. This fixed configuration is more complicated, since there is a 20 MHz channel at 5 GHz shared by two APs. The testing scenarios that we constructed will push the systems to their limits and we should be able to observe how different systems allocate the limited resources to best serve the wireless LAN clients. Our initial testing as we were setting up and calibrating the system confirmed that while this channel configuration did materially decrease the overall absolute capacity of the network, relative performance across vendors was maintained. High Density Wireless LANs: A Comparison 8

10 Client Configuration The clients were arranged in rows of 20. When a client machine was entered into the test bed, it was assigned a specific location, and named after that location; that never changed. For example, D10 is a Dell laptop running Windows 7 with a 3 stream n Wi-Fi network adapter. That machine has a physical label D10 and a software label D10 and would always be at location row D, seat 10. Figure 6 Client Devices in Seats Figure 7 shows the arrangement of clients on the floor. We ran each set of Chariot tests with three different client configurations - All, Half, and Third. When configuring the tests for fewer clients, we attempted to maintain an even spread of clients throughout the space. For the Half load configuration, we removed every other row of clients from the test. For the Third load configuration, we removed more rows as shown in Figure 7 below. Figure 7 Client Configurations High Density Wireless LANs: A Comparison 9

11 Comparison of High Density Testing We replicated the exact same network configuration with systems from Aruba and Cisco. We used a high availability configuration with two WLAN controllers and used their latest dual band, three stream indoor enterprise APs. Everything was mounted in the same locations as the Juniper gear. We configured the networks according to the manufacturer s recommendations for high density deployments. However, we did configure the same static channel plan that we created for the Juniper testing. The APs are mounted in the same locations and set to the same channels and channel sizes. All user devices were in the same locations. We were not able to get the Cisco network to complete the testing successfully for a significant number of tests. We had a very high number of Chariot initialization errors. Many clients dropped out before the tests began. When we did complete a Chariot test run, the throughput results were often very low. When large numbers of clients do not complete their streams in the allotted time for the test, the Chariot error processing at the end of test can be very long - much longer than a successful test. We were able to complete only one set of tests for the Cisco configuration. At the time of the testing, without complete analysis of the data, we did not fully understand the results and since the result of the Cisco test was impractically long test times, we abandoned further Cisco network testing. As we finished the test program and conducted the full post-test analysis, we decided that the partial Cisco test results told an important story - but not a story directly comparable to that which we are able to construct from the full suite of tests conducted on the Aruba and Juniper equipment. We report the Cisco results separately for this reason. For both the Aruba and Juniper systems, we were able to configure the system properly and we were able to get consistent behavior and reasonable throughput. We ran through the complete suite of tests with Aruba and Juniper including all client configurations. Out of the mountain of data we collected for Aruba and Juniper we believe that there are three themes are of particular interest: The comparison of aggregate throughput capacity; The comparison of MAC level frame retry rates - often the WLAN canary in the coal mine first indicator of underlying wireless LAN system issues; and The comparison of distribution of capacity between all the independent streams to various clients how equitably the capacity is shared under high load. Comparison of Aggregate Capacity Let's first look at the overall throughput capacity under the three types of traffic. Overall throughput is a measure of the total capacity of the network under a given load... without consideration of how that capacity is shared among all the user devices.we are purely looking at total capacity. Under the low, throttled throughput model, Juniper performs better at low client levels and maintains overall throughput as the number of client increases. Aruba, surprisingly, has rather low overall throughput with low numbers of clients but then matches Juniper as the number of clients increases. High Density Wireless LANs: A Comparison 10

12 600" 500" 400" Low%Throughput:%%Aruba%and%Juniper% Mbps% 300" 200" 100" Aruba" Juniper" 0" 0" 100" 200" 300" 400" Number%of%Clients% Figure 8 Under bidirectional traffic, Juniper and Aruba have very similar capacities, though Juniper has modestly better overall capacity at higher client levels. Bidirec5onal%Throughput:%Aruba%and% Juniper% 600.0# 500.0# 400.0# Mbps% 300.0# 200.0# 100.0# 0.0# 0# 100# 200# 300# 400# Number%of%Clients% Figure 9 High Density Wireless LANs: A Comparison 11

13 Under a purely downstream load, Juniper materially outperforms Aruba - particularly as the number of clients grows. Down%Throughput:%Aruba%and%Juniper% 600.0# 500.0# 400.0# Mbps% 300.0# 200.0# 100.0# 0.0# 0# 100# 200# 300# 400# Number%of%Clients% Figure 10 It is fair to create a composite measurement of these three traffic models, since in practice, aggregate load will be a composite of these traffic models. If we average the three measurements, we have a composite comparison in which at low numbers of clients, Aruba and Juniper have similar capacities. However, with increasing numbers of clients, the Aruba system slowly degrades in overall capacity while the Juniper system essentially maintains - modestly increasing and then decreasing in capacity # 500.0# 400.0# Composite%Throughput:%Aruba%and% Juniper% Mbps% 300.0# 200.0# 100.0# 0.0# 0# 100# 200# 300# 400# Number%of%Clients% Figure 11 As we can see from this composite measurement, Juniper not only has a higher total capacity, but reaches peak system capacity at about 50% more clients than Aruba. High Density Wireless LANs: A Comparison 12

14 Comparison of Retry Rates One of the key measures of underlying WLAN network issues is the MAC frame retry rate - how often MAC frames are retransmitted due to error or congestion. We looked at three examples (from the many tests) to get the 5 GHz MAC frame retry rates from the packet capture data: Bidirectional traffic with full client load, Bidirectional traffic with half client load and Low throughput with low client load. In every case, we can see dramatically higher frame retry rates in Aruba over Juniper. On average, Aruba consistently is about 1.9x higher in frame retry rates at 5 GHz. Junipers lower frame retry rates generally imply greater underlying stability and robustness. 60%# 50%# Bidirec4onal&Retry&Rate:&&302& Clients&;&Aruba&and&Juniper& Retry&Rate& 40%# 30%# 20%# 10%# 0%# 36# 44# 149# 157# 165# 5&GHz&Channel& Figure 12 For the bidirectional traffic model with full client load, we see Aruba has substantially higher frame retry rates than Juniper in 4 out of the 5 channels configured. High Density Wireless LANs: A Comparison 13

15 Birec3onal&Retry&Rate:&156&Clients&9& Aruba&and&Juniper& 60%# 50%# Retry&Rate& 40%# 30%# 20%# 10%# 0%# 36# 44# 149# 157# 165# 5&GHz&Channel& Figure 13 For the bidirectional traffic model with half client load (156 clients), Aruba has higher retry rates than Juniper in all 5 channels configured. 60%# 50%# Low&Retry&Rate:&95&Clients&7&Aruba& and&juniper& Retry&Rate& 40%# 30%# 20%# 10%# 0%# 36# 44# 149# 157# 165# 5&GHz&Channel& Figure 14 For the last case, with the low throttled load model and low client load (95 clients), Juniper consistently has a lower retry rate on all configured channels. High Density Wireless LANs: A Comparison 14

16 Comparison of Test Streams In our tests, each device has at least one TCP communications stream (generally down from the Chariot server to the device), and some have multiple streams - either an additional stream down to the device or, in the bidirectional test, an upstream stream from the device to the server. It is illustrative to look at the performance of the each network in terms of the histogram of results over all the clients, as well as the summary metrics of the average speed of each stream under each type of test, the standard deviation of the range of results, and particularly the frequency of occurrence of stream failure. Stream failure is defined as a stream that fails to report results by the end of the 120 second test. We presume zero throughput for that stream since no results were reported. The number of such failed streams is a measurement of how equitable and stable the system provides bandwidth to the set of clients. We will see large differences between Aruba and Juniper, particularly in stream failure rate. And when we consider our partial Cisco testing results, the issue of stream failure is even more compelling. Average Stream Throughput Let's first look at the average stream throughput rate as we vary the number of clients and the type of traffic between our low, bidirectional and downstream tests. Down%Stream%Throughput:%% Aruba%and%Juniper% Mbps% 4.00# 3.50# 3.00# 2.50# 2.00# 1.50# 1.00# 0.50# 0.00# 0# 100# 200# 300# 400# Number%of%Clients% Figure 15 For our downstream only tests, Aruba degrades noticeably more rapidly than Juniper, reflecting the aggregate capacity analysis, in this case, simply divided by the number of streams. High Density Wireless LANs: A Comparison 15

17 Bidirec5onal%Stream% Throughput:%Aruba%and%Juniper% Mpbs% 4.00# 3.50# 3.00# 2.50# 2.00# 1.50# 1.00# 0.50# 0.00# 0# 100# 200# 300# 400# Number%of%Clients% Figure 16 Similarly, the average bidirectional stream throughput mirrors the aggregate capacity for this test, as does the low, throttled stream throughput. Low%Stream%Throughput:%Aruba% and%juniper% Mbps% 4.00# 3.50# 3.00# 2.50# 2.00# 1.50# 1.00# 0.50# 0.00# 0# 100# 200# 300# 400# Number%of%Clients% Figure 17 We can see that in the low and downstream tests, Juniper usually delivers more average throughput per stream than Aruba, and for the bidirectional test, both systems were about the same for all numbers of clients. If we create a composite average flow, it shows that Juniper has a 12% better average stream throughput than Aruba over this range of client load. High Density Wireless LANs: A Comparison 16

18 Composite%Stream%Throughput:% Aruba%and%Juniper% Mbps% 4.00# 3.50# 3.00# 2.50# 2.00# 1.50# 1.00# 0.50# 0.00# 0# 100# 200# 300# 400# Number%of%Clients% Figure 18 Stream Histograms We analyzed the histogram distribution of the stream throughput for each vendor, under the three load scripts and client loads. 35%# 30%# 25%# 20%# 15%# 10%# 5%# 0%# Downstream%Traffic%2%%302%Client% Load% 0# 1# 2# 3# 4# 5# 6# 7# 8# 9# 11# 13# 15# 17# Mbps% Bidirec,onal%Traffic%3%302%Client%Load% 40.00%$ 35.00%$ Juniper$ 30.00%$ Aruba$ 25.00%$ 20.00%$ 15.00%$ 10.00%$ 5.00%$ 0.00%$ 0$ 0.5$ 1$ 1.5$ 2$ 2.5$ 3$ 3.5$ 4$ 4.5$ 5$ 5.5$ 6$ 6.5$ 7$ 7.5$ 8$ 8.5$ 9$ 10$ Mbps% 40%# 35%# 30%# 25%# 20%# 15%# 10%# 5%# 0%# Low%Traffic%.%302%Client%Load% 0# 0.5# 1# 1.5# 2# 2.5# 3# 3.5# 4# 4.5# 5# Mbps% Figure 19 Figure 20 Figure 21 30%# 25%# Downstream%Traffic%2%156%Client% Load% 30%# 25%# Bidirec,onal%Traffic%3%156%Client%Load% 35%# 30%# 25%# Low%Traffic%.%156%Client%Load% 20%# 20%# 20%# 15%# 15%# 15%# 10%# 10%# 10%# 5%# 5%# 5%# 0%# 0# 1# 2# 3# 4# 5# 6# 7# 8# 9# 11# 13# 15# 17# 19# 21# 23# Mbps% 0%# 0# 1# 2# 3# 4# 5# 6# 7# 8# 9# 11# 13# 15# 17# 19# 21# 23# Mbps% 0%# 0# 0.5# 1# 1.5# 2# 2.5# 3# 3.5# 4# 4.5# 5# Mbps% Figure 22 Figure 23 Figure 24 High Density Wireless LANs: A Comparison 17

19 Downstream%Traffic%2%95%Client%Load% Bidirec,onal%Traffic%3%95%Client%Load% Low%Traffic%.%95%Client%Load% 30%# 25%# 20%# 15%# 10%# 5%# 30%# 25%# 20%# 15%# 10%# 5%# 40%# 35%# 30%# 25%# 20%# 15%# 10%# 5%# 0%# 0# 1# 2# 3# 4# 5# 6# 7# 8# 9# 11# 13# 15# 17# 19# 21# 23# Mbps% 0%# 0# 1# 2# 3# 4# 5# 6# 7# 8# 9# 11# 13# 15# 17# 19# 21# 23# 25# 27# 29# Mbps% 0%# 0# 0.5# 1# 1.5# 2# 2.5# 3# 3.5# 4# 4.5# 5# Mbps% Figure 25 Figure 26 Figure 27 Over all traffic models and over all client loads, we can see the pattern that Aruba has more streams with zero (or very low) measurable bandwidth while Juniper tends to have a tighter (and higher throughput) clustering of bandwidth distribution between streams. The stream histogram data demonstrates that Juniper delivers a more equitable and predictable user experience there is less variation of the user experience between user devices in the same room than the Aruba network demonstrated. And as we shall see, the Cisco network could not be tested in this way since its highly unpredictable performance prevented full testing. Standard Deviation of Stream Throughput We saw common themes in the histograms, now lets see some summary data that illustrates some of the differences between these systems. One indicator is the standard deviation of the collection of the streams during a test. A lower standard deviation is an indicator of a more consistent and equitable distribution of throughput amongst the clients - a higher standard deviation indicates a less equitable distribution # Down%StreamThroughput% StDev% 8.00# Mbps% 6.00# 4.00# 2.00# 0.00# 0# 100# 200# 300# 400# Number%of%Clients% Figure 28 The higher standard deviation for the downstream throughput test suggests that a Aruba has a much less equitable distribution of throughput at low client load than Juniper, closing the gap as the number of clients increases. High Density Wireless LANs: A Comparison 18

20 Birdirec5onal%Stream% Throughput%StDev% Mbps% 10.00# 8.00# 6.00# 4.00# 2.00# 0.00# 0# 100# 200# 300# 400# Number%of%Clients% Figure 29 This pattern of unequal distribution (higher standard deviation) of bandwidth at low client load, that becomes more equal (lower standard deviation) at high load is repeated for the bidirectional stream tests. In both bidirectional and downstream cases, Aruba has substantially higher standard deviation in stream throughput for low numbers of clients and incrementally better (lower) standard deviation at high numbers of clients. In both systems, the standard deviation decreases with increasing numbers of clients # 8.00# Low%Stream%Throughput% StDev% Mbps% 6.00# 4.00# 2.00# 0.00# 0# 100# 200# 300# 400# Number%of%Clients% Figure 30 For the low throughput test, both systems deliver very similar standard deviations in stream throughput. Juniper delivers a more consistent (more equitable) variation in flow throughput with variation between flows remaining more consistent (and delivering a more consistent user experience) as the number of clients increases. High Density Wireless LANs: A Comparison 19

21 Stream Failure Rate Perhaps the most revealing measurement from the stream data comes from the stream failure data. While both systems showed an increased stream failure with increasing client load, Aruba has a dramatically higher stream failure that appears to have an inflection point around 100 client stations for all three traffic models. 30%# 25%# Down%Stream%Failure%Rate% %age%streams% 20%# 15%# 10%# 5%# 0%# 0# 100# 200# 300# 400# Number%of%Clients% Figure 31 Bidirec7onal%Stream%Failure% Rate% 30%# %age%streams% 25%# 20%# 15%# 10%# 5%# 0%# 0# 100# 200# 300# 400# Number%of%Clients% Figure 32 High Density Wireless LANs: A Comparison 20

22 Low%Stream%Failure%Rate% 30%# 25%# %age%streams% 20%# 15%# 10%# 5%# 0%# 0# 100# 200# 300# 400# Number%of%Clients% Figure 33 %age%streams% Composite%Stream%Failure% Rate% 30%# 25%# 20%# 15%# 10%# 5%# 0%# 0# 100# 200# 300# 400# Number%of%Clients% Figure 34 For all types of traffic, Aruba has a materially higher rate of stream failures - clients that receive no data throughput - than Juniper. This stream failure rate dramatically increases as the number of clients increases from 95 to 156 and continues to increase to the full client load of 302. Juniper has a much lower and much more stable stream failure rate through 156 clients, before beginning an increase in stream failure at full client load. High Density Wireless LANs: A Comparison 21

23 A Partial Analysis of Cisco Results As previously noted, we were not able, within the time constraints of this test, to fully evaluate the Cisco configuration. We found high failure rates of Chariot test streams which slowed the testing process unacceptably. And as we have seen above - failure rates of the Chariot test streams is an important distinguishing factor between Aruba and Juniper. However, the data from the one complete test of the Cisco configuration is interesting and we think illustrative. This is for the downstream only, full throughput traffic model with the full 302 client load. 70.0%$ Cisco%Downstream%302%Clients%Stream% Distribu:on% 60.0%$ 50.0%$ %age%streams% 40.0%$ 30.0%$ 20.0%$ 10.0%$ 0.0%$ 0$ 1$ 2$ 3$ 4$ 5$ 6$ 7$ 8$ 9$ 10$ 11$ 12$ 13$ 14$ 15$ 16$ 17$ 18$ 19$ 20$ Mbps% Figure 35 As we can see in Figure 35, the histogram of the distribution of stream throughputs shows that 62% of the streams had no measurable data throughput. The aggregate throughput of these test runs was 376 Mbps which is OK, but there is a highly inequitable distribution of that capacity. About 3% of the streams had throughput higher than 10 Mbps. These few streams constituted the bulk of the data throughput while the majority of the devices under test were completely starved of data capacity. For similar tests, Aruba had a stream error rate of 17% and Juniper s error rate was 11%. The 62% stream error rate for Cisco on this test illustrates why we were unable to complete the testing for Cisco. For the other tests that we attempted, we could see the traffic generated over the air but the error rate was so high that Chariot did not report any useful results. While we do not have the measurements for the other traffic tests and other client load models, we suspect that the same pattern would ensue based on our difficulties in getting these tests to run reliably for the Cisco network. High Density Wireless LANs: A Comparison 22

24 Conclusions Almost all enterprise wireless LANs will have some high density areas in their networks and these areas are often in visible and highly used portions of the network auditoriums, classrooms, conference areas, etc.. We found meaningful and compelling differences between the network performance delivered by three important wireless LAN vendors: Aruba, Cisco and Juniper. Key Findings We are measuring the limits of capacity of each of these networks under high user density load. Each network was clearly at maximum load, as measured by the degradation of aggregate throughput and increased MAC frame retry rates as more client devices were added. Juniper maintained consistent performance as load increased from under 100 clients to over 300 clients delivering not only the highest aggregate throughput, but the lowest MAC frame retry rates and the most stability under load as measured by the number of simultaneous streams with real, non-zero throughput. Its maximum performance was at over 50% more clients than Aruba s maximum performance and at client and traffic loads at which the Cisco network could not be successfully tested. Aruba delivered materially lower performance than Juniper not only lower throughput but, more importantly, at lower numbers of clients. We noted that the Aruba system had materially higher MAC frame retry rates (1.9x) and materially higher data stream failures (about 2x on average) than Juniper. The higher stream failure rates indicate that many more clients were starved for data throughput capacity under the Aruba system than the Juniper system.the Aruba system did successfully complete the full range of tests. The Cisco network was challenged by this test. We had great difficulty running the tests due to the very high number of stream failures that substantially increased the time required to run the test. With limited availability of the facility, we were unable to complete a full suite of tests for the Cisco network. However, the one completed test is suggestive. Under a maximum client load of 302 clients, under the downstream only traffic load, the Cisco network had over 62% stream failures that is, almost two-thirds of the test throughput streams had zero measurable data throughput. This compares to 17% for Aruba and 11% for Juniper. Both the Juniper and Aruba networks delivered a useful network under these conditions of high user density - unlike the Cisco network which did not deliver a reliable, robust network. The Juniper network delivered the most capable network of demonstrably higher capacity, greater equity of throughput between user devices, and of higher stability as the network load increased. High Density Wireless LANs: A Comparison 23

25 Appendix A About Novarum Novarum is an independent consulting firm specializing in wireless broadband technology and business. Novarum provides consulting, strategic advice, analysis and network design for cities, service providers, enterprises and vendors in the wireless broadband industry. Our technology focus spans Wi-Fi, WiMAX and 4G cellular data systems. Novarum offers a unique insider perspective from pioneers in the wireless and networking industry who have practical experience bringing wireless products to market. Phil Belanger Phil has over 25 years of broad leadership in the technology, marketing and standards of data networks. Phil pioneered local area networking technology with Zilog and Corvus and extended that leadership by co-leading the multi-company technical and marketing efforts that produced the original IEEE wireless LAN standard. Phil defined the original market position of wireless LANs for mobile computing with Xircom. While at Aironet, he broadened the market for wireless LANs and laid the foundation for Wi-Fi's success with the acquisition of Aironet by Cisco. Phil was one of the founders of the the Wi-Fi Alliance and served as the group s initial Chairman, creating the Wi-Fi brand and promoting Wi-Fi for the entire industry. He helped create the business model for Wi-Fi service providers with Wayport and expanded the market for Wi-Fi infrastructure with extended range technology of Vivato and municipal mesh networks at BelAir Networks. Ken Biba Ken is a rocket scientist. He also has many years experience in the network information systems industry bringing a unique background of general management with a strong product and marketing focus in network systems and information security. Ken was an early engineer of the Internet in He has co-founded and managed four notable networking companies Sytek, which was focused on cable TV-based local and metropolitan data networks, Agilis which was focused on wireless handheld computers, Xircom, which developed local area network client products for mobile computing, and Vivato, which was focused on scaling Wi-Fi infrastructure to cover campuses and metropolitan areas. Ken's perspective as CEO, board member of public and private companies, and as a technologist brings unique insight to the business, market and technology of bringing useful wireless solutions to users. Ken has a Bachelor of Science in Physics (Magna Cum Laude, Tau Beta Pi) and a Master of Science in Computer Science from Case Western Reserve University. Wayne Gartin Wayne is a senior executive with world-wide experience at start-ups and Fortune 500 companies. He has built high level relationships and delivered business partnerships at all levels for companies in the communication, software, and semiconductor markets. Wayne has worked with industry leading suppliers in all aspects of network technology, including long haul transport, metropolitan networks, wired and wireless LANs. He has successfully run multi-million dollar sales teams for companies in the access (last mile) consumer oriented markets, Passive Optical Networks, VoIP, and IMS. Wayne has held executive and senior level positions at Centillium, Agility (now JDSU), Bandwidth 9 (now NeoPhotonics), Infineon, Lucent, Adaptec, and Intel. He is also the co-founder of a semi-conductor IP company. Wayne s experience with multiple channels and leading successful sales teams to multi-million dollar revenue levels brings a unique insight to the strategies necessary to successfully launch new products and technologies into the market. Wayne has a BS in Math and an MBA from the University of Utah. He has been a certified instructor for sales and marketing courses in strategic planning, negotiations, and sales management. High Density Wireless LANs: A Comparison 24

26 Appendix B Aruba Configuration version 6.2 hostname "Aruba3600-1" clock timezone 0 location "Building1.floor1" controller config 1 ip NAT pool dynamic-srcnat ip access-list eth validuserethacl permit any netservice svc-netbios-dgm udp 138 netservice svc-snmp-trap udp 162 netservice svc-pcoip2-tcp tcp 4172 netservice svc-syslog udp 514 netservice svc-l2tp udp 1701 netservice svc-ike udp 500 netservice svc-smb-tcp tcp 445 netservice svc-citrix tcp 2598 netservice svc-dhcp udp alg dhcp netservice svc-https tcp 443 netservice svc-pptp tcp 1723 netservice svc-ica tcp 1494 netservice svc-sccp tcp 2000 alg sccp netservice svc-http-accl tcp 88 netservice svc-telnet tcp 23 netservice svc-netbios-ssn tcp 139 netservice svc-sip-tcp tcp 5060 netservice svc-kerberos udp 88 netservice svc-tftp udp 69 alg tftp netservice svc-http-proxy3 tcp 8888 netservice svc-noe udp alg noe netservice svc-cfgm-tcp tcp 8211 netservice svc-adp udp 8200 netservice svc-pop3 tcp 110 netservice svc-pcoip-tcp tcp netservice svc-pcoip-udp udp netservice svc-lpd-tcp tcp 631 netservice svc-rtsp tcp 554 alg rtsp netservice svc-msrpc-tcp tcp netservice svc-dns udp 53 alg dns netservice vnc tcp netservice svc-h323-udp udp netservice svc-h323-tcp tcp 1720 netservice svc-vocera udp 5002 alg vocera netservice svc-http tcp 80 netservice svc-http-proxy2 tcp 8080 netservice svc-sip-udp udp 5060 netservice svc-nterm tcp High Density Wireless LANs: A Comparison 25

27 netservice svc-noe-oxo udp 5000 alg noe netservice svc-natt udp 4500 netservice svc-ftp tcp 21 alg ftp netservice svc-microsoft-ds tcp 445 netservice svc-svp 119 alg svp netservice svc-smtp tcp 25 netservice svc-gre 47 netservice web tcp list "80 443" netservice svc-netbios-ns udp 137 netservice svc-sips tcp 5061 alg sips netservice svc-smb-udp udp 445 netservice svc-cups tcp 515 netservice svc-esp 50 netservice svc-v6-dhcp udp 546 netservice svc-snmp udp 161 netservice svc-bootp udp netservice svc-pcoip2-udp udp 4172 netservice svc-msrpc-udp udp netservice svc-ntp udp 123 netservice svc-icmp 1 netservice svc-ssh tcp 22 netservice svc-lpd-udp udp 631 netservice svc-v6-icmp 58 netservice svc-http-proxy1 tcp 3128 netservice svc-vmware-rdp tcp 3389 netexthdr default ip access-list session control ip access-list session allow-diskservices ip access-list session v6-icmp-acl ip access-list session validuser network any any deny any any any permit ipv6 any any any permit ip access-list session vocera-acl ip access-list session v6-https-acl ip access-list session vmware-acl ip access-list session v6-control ip access-list session icmp-acl High Density Wireless LANs: A Comparison 26

28 ip access-list session testing ip access-list session captiveportal ip access-list session v6-dhcp-acl ip access-list session allowall ip access-list session v6-dns-acl ip access-list session https-acl ip access-list session sip-acl ip access-list session ra-guard ip access-list session dns-acl ip access-list session citrix-acl ip access-list session tftp-acl ip access-list session skinny-acl ip access-list session srcnat ip access-list session vpnlogon ip access-list session logon-control ip access-list session allow-printservices ip access-list session v6-allowall ip access-list session cplogout ip access-list session http-acl ip access-list session dhcp-acl ip access-list session v6-http-acl ip access-list session captiveportal6 ip access-list session ap-uplink-acl ip access-list session noe-acl High Density Wireless LANs: A Comparison 27

29 ip access-list session svp-acl ip access-list session ap-acl ip access-list session v6-ap-acl ip access-list session h323-acl ip access-list session v6-logon-control aaa derivation-rules user test vpn-dialer default-dialer ike authentication PRE-SHARE aea06b09f946b8ead663bb1b77b7edc345861acb504d2749 user-role ap-role user-role guest-logon user-role guest user-role stateful-dot1x user-role logon controller-ip vlan 1 interface mgmt shutdown dialer group evdo_us init-string ATQ0V1E0 dial-string ATDT#777 dialer group gsm_us init-string AT+CGDCONT=1,"IP","ISP.CINGULAR" dial-string ATD*99# dialer group gsm_asia init-string AT+CGDCONT=1,"IP","internet" dial-string ATD*99***1# dialer group vivo_br High Density Wireless LANs: A Comparison 28

30 init-string AT+CGDCONT=1,"IP","zap.vivo.com.br" dial-string ATD*99# vlan 2 spanning-tree mode rapid-pvst no spanning-tree spanning-tree vlan 1 spanning-tree vlan 2 interface gigabitethernet 1/0 description "GE1/0" trusted trusted vlan switchport mode trunk interface gigabitethernet 1/1 description "GE1/1" shutdown trusted trusted vlan switchport mode trunk interface gigabitethernet 1/2 description "GE1/2" shutdown trusted trusted vlan interface gigabitethernet 1/3 description "GE1/3" shutdown trusted trusted vlan interface vlan 1 ip address High Density Wireless LANs: A Comparison 29

31 interface vlan 2 ip address master-redundancy master-vrrp 10 peer-ip-address ipsec f d664d2b80e9fd895f9cedc4126e0dd4b6797 vrrp 10 priority 120 ip address description "Preferred Master" vlan 1 preempt delay 0 tracking master-up-time 30 add 20 no shutdown ip default-gateway uplink disable ap mesh-recovery-profile cluster RecoverytyH0K51Ex/NTel5u wpa-hexkey dc822887ebd9dbeffb4b9443c97624de7e6582f43985b8d30eaefb5b09b2722f050b5ef0a949d2f0f9c38053b5b5b391a06eeb524f947ac84ec0c5 48abc88d cea eae8e734eb2dae2 crypto isakmp policy 20 encryption aes256 crypto ipsec transform-set default-boc-bm-transform esp-3des esp-sha-hmac crypto ipsec transform-set default-rap-transform esp-aes256 esp-sha-hmac crypto ipsec transform-set default-aes esp-aes256 esp-sha-hmac crypto dynamic-map default-dynamicmap set transform-set "default-transform" "default-aes" crypto isakmp eap-passthrough eap-tls crypto isakmp eap-passthrough eap-peap crypto isakmp eap-passthrough eap-mschapv2 vpdn group l2tp ip dhcp excluded-address ip dhcp excluded-address ip dhcp excluded-address ip dhcp pool test default-router dns-server lease network High Density Wireless LANs: A Comparison 30

32 authoritative service dhcp vpdn group pptp tunneled-node-address adp discovery enable adp igmp-join enable adp igmp-vlan 0 ap ap-blacklist-time 3600 mgmt-user admin root d2455a780128b754827b616d962e664a4b375ae2d66f17e88d database synchronize period 2 database synchronize rf-plan-data ip mobile domain default ip igmp ipv6 mld no firewall attack-rate cp 1024 ipv6 firewall ext-hdr-parse-len 100 firewall cp packet-capture-defaults tcp disable udp disable interprocess disable sysmsg disable other disable ip domain lookup country US High Density Wireless LANs: A Comparison 31

33 aaa authentication mac "default" aaa authentication dot1x "default" max-authentication-failures 5 no validate-pmkid termination eap-type eap-peap termination inner-eap-type eap-mschapv2 no cert-cn-lookup aaa authentication dot1x "default-psk" max-authentication-failures 5 no validate-pmkid termination enable termination eap-type eap-peap termination inner-eap-type eap-mschapv2 no cert-cn-lookup aaa authentication-server radius " " host " " key c17f62b f1200ca80783e04c6effddac8c793 timeout 8 use-ip-for-calling-station aaa server-group "default" auth-server aaa server-group "test" auth-server aaa profile "default" authentication-dot1x "default-psk" dot1x-server-group "default" radius-accounting "default" aaa profile "default-dot1x-psk" authentication-dot1x "default" dot1x-server-group "default" radius-accounting "default" no wired-to-wireless-roam no devtype-classification aaa authentication captive-portal "default" aaa authentication wispr "default" aaa authentication vpn "default" aaa authentication vpn "default-rap" High Density Wireless LANs: A Comparison 32

34 aaa authentication mgmt enable aaa authentication stateful-ntlm "default" aaa authentication stateful-kerberos "default" aaa authentication stateful-dot1x enable aaa authentication wired web-server session-timeout 3600 guest-access- aaa password-policy mgmt control-plane-security no cpsec-enable auto-cert-prov ids wms-general-profile poll-retries 3 ids wms-local-system-profile valid-network-oui-profile qos-profile "default" policer-profile "default" ap system-profile "default" lms-ip lms-preemption lms-hold-down-period 5 ap regulatory-domain-profile "default" country-code US valid-11g-channel 1 valid-11g-channel 6 valid-11g-channel 11 valid-11a-channel 36 valid-11a-channel 40 valid-11a-channel 44 High Density Wireless LANs: A Comparison 33

35 valid-11a-channel 48 valid-11a-channel 149 valid-11a-channel 153 valid-11a-channel 157 valid-11a-channel 161 valid-11a-channel 165 valid-11g-40mhz-channel-pair 1-5 valid-11g-40mhz-channel-pair 7-11 valid-11a-40mhz-channel-pair valid-11a-40mhz-channel-pair valid-11a-40mhz-channel-pair valid-11a-40mhz-channel-pair ap wired-ap-profile "default" ap enet-link-profile "default" ap mesh-ht-ssid-profile "default" ap lldp med-network-policy-profile "default" ap mesh-cluster-profile "default" ap lldp profile "default" ap mesh-radio-profile "default" ap wired-port-profile "default" ids general-profile "default" ids unauthorized-device-profile "default" ids profile "default" rf arm-profile "default" assignment disable rf arm-profile "disabled" assignment disable rf optimization-profile "default" rf event-thresholds-profile "default" rf am-scan-profile "default" rf dot11a-radio-profile "AP1" High Density Wireless LANs: A Comparison 34

36 channel 165 spectrum-load-balancing spectrum-load-bal-domain "dome" arm-profile "disabled" rf dot11a-radio-profile "AP2" channel 36+ spectrum-load-balancing spectrum-load-bal-domain "dome" rf dot11a-radio-profile "AP3" channel 44+ spectrum-load-balancing spectrum-load-bal-domain "dome" arm-profile "disabled" rf dot11a-radio-profile "AP4" channel 149+ spectrum-load-balancing spectrum-load-bal-domain "dome" arm-profile "disabled" rf dot11a-radio-profile "AP5" channel 157+ spectrum-load-balancing spectrum-load-bal-domain "dome" arm-profile "disabled" rf dot11a-radio-profile "AP6" channel 165 spectrum-load-balancing spectrum-load-bal-domain "dome" arm-profile "disabled" rf dot11a-radio-profile "default" spectrum-load-balancing beacon-regulate arm-profile "disabled" rf dot11g-radio-profile "AP1-24ghz" channel 1 spectrum-load-balancing rf dot11g-radio-profile "AP2-24ghz" channel 6 spectrum-load-balancing rf dot11g-radio-profile "AP3-24ghz" High Density Wireless LANs: A Comparison 35