Multi-client Emulation Platform for Performance Testing of High Density 802.11 Networks

Multi-client Emulation Platform for Performance Testing of High Density 802.11 Networks G. Álvarez 1, M. Álvarez 1, F. Bagalciague 1, G. Capdehourat 1, J. González 1, J. Marfia 1, P. Porteiro 1 and A. Rodríguez 1 Abstract During the last years the world has witnessed the great success of the IEEE 802.11 standard with an exponential increase in the number of Wi-Fi networks deployed around the globe. This fact was also accompanied by a great evolution of the technology, originally created for a few clients and today supporting thousands of users in a single wireless LAN. One of the most important verticals for Wi-Fi today is education, with several one-to-one programs running around the world, such as the nationwide Plan Ceibal in Uruguay. The present paper describes the development of an 802.11 portable multi-client testing platform, which enables the emulation of a typical high density classroom scenario to test the wireless infrastructure. The prototype was successfully validated for two different laptop models used by Plan Ceibal, considering both throughput and lower network layers performance. The results indicate this might be the first step towards an open platform to enable active in-site Wi-Fi performance testing for large scale scenarios. Index Terms Wi-Fi, client emulation, performance testing. I. INTRODUCTION In the past decade, the great success of the IEEE 802.11 standard, popularly known as Wi-Fi, has been one of the key factors that has driven the massive deployment of wireless Internet access worldwide. The technology that had born to provide a wireless solution for small-size LAN environments evolved to support thousands of clients in a single WLAN (e.g. large conferences or stadiums [1]). Moreover, end user devices have improved as well, and cheaper Wi-Fi capable terminals are developed every day. In this context, and considering the proved importance of broadband access in the countries development [2], [3], ICT proposals for education have begun to take shape in real deployments, such as the pioneer Plan Ceibal [4] in Uruguay, Conectar Igualdad in Argentina [5] and ConnectED in the US [6], among others. The novel one-to-one educational model consists of provisioning each student with a computer and also providing wireless Internet access at every school. This new paradigm resulted in a tough technical challenge for 802.11-based solutions, which evolved from a coverage-driven Wi-Fi design to a capacity-based approach, particularly focused on the quality of experience (QoE) [7] of students and teachers. Although a first theoretical analysis is useful to define the proper design criteria, then extensive real-world testing is necessary to support the selected criteria. On the other hand, after the design and deployment phase, a suitable post-installation validation should be done at each school, in order to check if the deployed infrastructure supports 1 Plan Ceibal, Avda. Italia 6201, Edificio Los Ceibos, 11500, Montevideo, Uruguay. Corresponding author: gcapdehourat at ceibal.edu.uy the previously defined requirements. This is actually the only way to certify that a certain school infrastructure was successfully deployed. A possible solution to test a Wi-Fi deployment is to conduct stress tests, analyzing the network performance in a high-density high-load situation. Thus, it is possible to determine for a certain Wi-Fi infrastructure which kind of applications it is able to support. One way to perform these tests is taking a large amount of devices (laptops, tablets) to the school, connecting them to the wireless network and use a traffic generation tool such as iperf [8] to test the network capacity. Different measurements can be used to determine the network performance, depending on the particular layer behavior to be analyzed. For example, both network and transport layer metrics can be obtained with iperf, while lower layers performance can be analyzed with wireless packet captures during the stress test. It is also possible to have suitable models to estimate relevant applications performance (e.g. education platforms, video streaming) from lower layers measurements or directly compute them by generating real application traffic. The previously described stress test is a suitable way to address the extensive real-world testing needs except for one point: carrying hundreds of laptops to schools each time you perform a test. This fact implies several inconvenient aspects: the logistics to transport hundreds of laptops and other equipment, tens of technicians and several hours of field work. A portable tool for 802.11 multi-client emulation would enable to perform these kind of analysis in a much more easy way and help to reduce costs and simplify the work of the technicians involved. Moreover, a platform for test automation would enable the possibility of performing more complex tests just changing the software application layer at the top of the emulator, also providing performance analytics. This work describes the prototype developed as a first step to reach the desired 802.11 muti-client emulator. The platform was validated for two different laptop models used by Plan Ceibal in the 2.4 GHz band, considering both throughput and air traffic captures. The results indicate that the prototype enables high-density Wi-Fi testing, accurately emulating the behavior of the laptops delivered by Plan Ceibal. This first version of the tool, not only proved to be a useful solution to enable in-site Wi-Fi testing, but also enables future customization for different purposes because of the system modular design. To the best of the authors knowledge this work is the first

one aimed in constructing a multi-client emulation platform for performance testing of 802.11-based networks. The typical research tools that have been used by the academia are simulation (e.g. ns3 [9]) or real-world testbeds (e.g. [10]), but none similar to this proposal which is argued to be more suitable for high-density emulation. Existing solutions from the industry 1 provide a complete system to properly test a Wi-Fi network. While these tools are useful for certain applications, none of them enables real RF-emulation of different devices, as the client hardware is not the same that user devices have. Thus, the results obtained with these tools may present great differences from the real performance that users will get in the same real-world environment. Moreover, this kind of solutions typically emulate the different RF channel between transmitter and receiver by simply using different power levels for the different clients, but still using the same RF chain and a single antenna for all of them. This way, the actual physical channel, determined by the locations of clients and APs, antennas position, orientation and polarization, is not properly emulated. As Wi-Fi beamforming technology evolves this variables seem to be too important to be ignored. The rest of the paper is organized as follows. The next section presents the requirements defined for the system implementation. Section 3 describes the prototype developed, highlighting the key points in the design procedure. In Section 4 the system validation tests are presented, discussing the accuracy obtained for each laptop model emulated. Finally, the last section comments the conclusions of the system implementation and discuss the future steps for reaching a mature product version to be used in the field. II. SYSTEM REQUIREMENTS Before selecting the particular hardware and software to build the prototype, several requirements were established for the system implementation: The system should be a cost-effective and portable solution for Wi-Fi performance testing. Each equipment should enable to emulate at least 20 end user devices (to represent a typical classroom size). The system should enable proper client MAC layer and RF emulation, with one complete Tx/Rx chain per user (one Wi-Fi card, cable, antenna and the connectors set per emulated device). The solution should emulate as accurate as possible the hardware and software of the laptop models delivered by Plan Ceibal. Similar test results for three selected indicators detailed in Section IV (average aggregated TCP throughput, total airtime consumption and effective data rate). The platform should be flexible to enable its future evolution (e.g. new functionalities such as different types of client traffic patterns and QoE estimation [11]). Two different laptops models were considered for the prototype implementation, Magallanes 3 (MG3) [12] and 1 For example IxVeriWave from Ixia or Landslide from Spirent. Fig. 1. Scheme of the system developed. Positivo BGH 11cle2 (11cle2) [13], which correspond to the ones that were delivered in the largest amount by Plan Ceibal to the students. The first one is an Intel Atom based laptop with 1 Gbyte of RAM and 8 GB of SSD storage. The wireless network interface controller (WNIC) is an Intel Wireless-N 100 mini PCI express radio card [14]. The most important details about the WNIC are: IEEE 802.11b/g/n Wi-Fi wireless adapter (only supports 2.4 GHz). MIMO 1x2:1, so it only supports one spatial stream. The maximum datarate for 802.11 physical layer is 72.2 Mbps @20 MHz channel width. The second model considered is an Intel Celeron based laptop with 2 Gbyte of RAM and 16 GB of SSD storage. The WNIC is an Intel Wireless-N 7260 mini PCI express radio card [15]. The most important details about the WNIC are: IEEE 802.11a/b/g/n Wi-Fi wireless adapter (dual-band 2.4 GHz and 5 GHz). MIMO 2x2:2, so it supports up to two spatial streams. The maximum datarate for 802.11 physical layer is 144.4 Mbps @20 MHz channel width. Both laptops models are provided to the students with Ubuntu [16], a Linux-based open source operating system. III. DESIGN AND DEVELOPMENT OF THE SYSTEM The system developed is depicted in Figure 1 and consists of a main computer with several Wi-Fi interfaces which are connected to their respective antennas. The final setup tested for the emulation of both laptop models consisted of 24 radios with their respective antennas. The system setup aims to substitute 24 laptops with one main box connected with 24 antennas to emulate the behavior of a classroom full of Plan Ceibal s users. Throughout the design and development

process several hardware and software issues were addressed which are summarized below. The first major problem was to integrate a large number of Wi-Fi WNICs into a single motherboard. The solution found consists of a motherboard with several PCI express ports 2 and then flexible PCI express to mini PCI express splitters 3. The development started with x4 splitters and then arrived to the final solution with x8 splitters, using 3 splitters to reach a total of 24 radios in a single emulator. The Wi-Fi WNICs selected were exactly the same as the ones integrated in each laptop model considered (Intel Wireless-N 100 and Intel Wireless-N 7260 respectively). A generic PC case was modified in order to make the prototype more portable for trials. Figure 2 shows the customized case for the main box. Because of the heterogeneity of laptop models and Wi-Fi technologies supported by students laptops, the prototype should enable the operation of 802.11abgn radios with two Tx/Rx chains. The RF behavior of Plan Ceibal laptops was characterized and customized antennas were designed and manufactured. The solution achieved consisted of 2.4GHz/5GHz MIMO antennas, with two RP-SMA male connectors and a 3 meter length coaxial cable RG174/U. This way it is possible to emulate the actual location of the students inside a classroom. The connection from mini PCI express WNICs to the antennas consist of U.FL to RP-SMA pigtails. The U.FL connector is the one included in the WNICs, while the RP-SMA is plugged to each of the cables attached to each antenna. As mentioned before, each antenna has two elements which are used in a different way depending on the WNIC. On the one hand the Intel Wireless-N 100 is MIMO 1x2:1, which means it only has one RF chain and only uses both elements for receive diversity. On the other hand the Intel Wireless-N 7260 is MIMO 2x2:2, so it has two RF chains and both elements are used for both transmit and receive multiplexing. To complete the main computer, in addition to the suitable motherboard with many PCI ports, the other standard elements were selected, such as microprocessor, RAM memory, hard disk and power source. The operating system was chosen in order to get the most similar behavior of Wi-Fi operation with respect to the laptops. Thus, in order to work with similar wireless drivers to the ones used by student devices, the same Linux-based system of the laptops was considered for the emulator, in this case Ubuntu. Another important component of the emulation platform is the corresponding software to automate the stress tests. A Python code was developed for this purpose, which manages all the steps involved in conducting the test. First, the script handles the automatic association to the desired wireless network, defined by the SSID and the WPA pre-shared key. Then, all the traffic generation for each wireless interface is prepared, which is synchronized with a local server connected via Ethernet to the access points conforming 2 A motherboard Asus Maximus VI Extreme was selected. 3 Provided by Amfeltec http://amfeltec.com/splitters/. Fig. 2. Customized case for the main box of the emulator and antennas. the wireless LAN. The application considered in this first prototype was the simultaneous download of large files by all wireless clients. To emulate this application, downlink TCP traffic flows were generated with iperf [8] from the local server. Finally, the corresponding test logs were processed in order to generate the final results, such as average total throughput and bandwidth sharing between clients. IV. VALIDATION TESTS AND RESULTS In order to validate the prototype developed, the same test procedure was conducted for both laptop models considered. The emulator was equipped on each case with the same Wi-Fi WNICs than each laptop, with a total number of 24 radio cards. As mentioned before, the application tested was simultaneous file downloading emulated with 5-minute-long TCP downlink traffic flows generated with iperf, using a mini PC as local server 4. The same application was tested with the real laptops and the corresponding emulated version developed, and results for both cases were compared. The access point (AP) used in the tests was a Cisco Aironet 2702 which supports the latest 802.11ac standard, although the tests were conducted with 802.11n because the laptops do not support 802.11ac. The tests were performed in a classroom located in an office environment, with interference conditions similar to those of a real school scenario. The antennas were disposed over the desktops in the same location as the laptops throughout the room, thus emulating the same client distribution for both cases. 4 A Giada mini PC was used: http://www.giadapc.com/products/minipc/.

The AP operated only in the 2.4 GHz band, in a 20 MHzwidth channel selected as the least occupied from the standard channels used by Wi-Fi in 2.4 GHz (1, 6 or 11). The spectrum analyzer used to monitor de RF environment during the tests was the WiSpy USB adapter with the Chanalizer software [17]. The spectrum monitoring was only considered to select the channel and to ensure not having significative differences between the external interference in both tests. In the next section all the measurements gathered during the tests considered for the validation purposes are presented. A. Measurements considered for the validation In addition to the spectrum monitoring, two type of measurements were considered to compare the behavior of real clients versus emulated clients. On the one hand, the application logs (in this case iperf logs) for each client were considered, in order to get higher network layer statistics from the test. On the other hand, air packets traffic captures were obtained with a specific device (in this case an Apple Macbook Pro) running the standard capture tool Wireshark [18]. This device was particularly selected to enable air captures for packets transmitted using multiple spatial streams, which is not possible with standard USB 802.11 adapters as they typically support only one spatial stream. From this packet captures, several statistics from lower network layers were computed in order to compare real versus emulated clients behavior. While the most important matchings desired were the ones in higher layers, such as total TCP throughput and bandwidth sharing between clients, the lower layer statistics comparison was also taken into account for the validation. Such information is also relevant to analyze how similar is the behavior between laptops and emulated clients. In order to validate the emulation, a maximum relative error of 20% was allowed for the selected indicators: average aggregated TCP throughput, total airtime consumption and effective data rate. The first one is the total amount of transferred data per time unit at the transport layer. The second one corresponds to the time percentage during which the medium is occupied by the AP or the clients. The last indicator corresponds to the average transmission data rates over the air weighted by the corresponding packet sizes. Although they were not used for validation purposes, other indicators were computed from test measurements. In addition to the total TCP throughput, the sharing between clients was studied analyzing the average throughput per client. The same procedure was conducted at the lower level with the airtime consumption per client. Finally, the time consumed by retries was also compared and the transmission data rates histogram was computed for all tests. In order to avoid transient and synchronization effects, the first and last minute from the 5-minute-length tests were discarded for computing performance metrics. This way the simultaneous stationary operation of all the clients was ensured during the interval considered. Thus, all airtime consumption measures are expressed as a percentage of the 3-minute-length subpart of the test. Fig. 3. Total TCP Throughput Average per client Laptops 16 0.7 0.50 Emulator 16 0.7 0.35 TABLE I THROUGHPUT RESULTS FOR EMULATION OF LAPTOPS MG3. Throughput TCP comparison for emulation of laptops MG3. B. Emulation results for laptops MG3 The validation tests for the laptop model MG3 were conducted with 23 clients, as one of the laptops failed at the moment of carrying on the tests. In order to have the same number of clients for the comparison, one of the emulator WNICs was turned off, to operate only with 23 clients also. The higher layer results, which correspond to iperf logs, are analyzed first. The total average results are summarized in Table I. The aggregated traffic was the same in both cases, while the variations between clients were larger for laptops than for the emulator. This result is ratified by Figure 3, which shows the average TCP throughput per client in ascending order for both laptops and emulator. While a similar general behavior is observed for both cases, it can be noticed that the laptops with larger throughputs are above the corresponding emulator clients, and the opposite happens with the laptops with smaller throughputs, which are below the emulator clients. There is no clear explanation of this bandwidth sharing difference, but one possible hypothesis is that the hardware variations among different laptops (even being all of the same model) could produce this effect, while the emulator clients are all over the same hardware, so there is no possible hardware variability in that case. Concerning air packet captures, the Table II resumes the main results. Again, a larger variation between clients is observed for the laptops, which is more clear looking at the airtime consumption per client shown in Figure 4. Most parameters are similar for both cases, with the exception of the airtime consumed by retries, which is larger in the emulator test. On the other hand, the airtime consumed by control frames is quite similar in both tests, with 24% for the laptops and 26% for the emulator. In Figure 5 the data rates histograms are compared,

Total Airtime Retries Average data rate Laptops 67.7 1.9 9 43.5 Emulator 70.1 1.1 14 41.8 TABLE II AIRTIME CONSUMPTION RESULTS FOR EMULATION OF LAPTOPS MG3. Total TCP Throughput Average per client Laptops 44.2 1.8 0.82 Emulator 52.0 2.2 0.46 TABLE III THROUGHPUT RESULTS FOR EMULATION OF LAPTOPS 11CLE2. Total Airtime Retries Average data rate Laptops 65.6 1.4 5 111 Emulator 73.6 0.7 4 118 TABLE IV AIRTIME CONSUMPTION FOR EMULATION OF LAPTOPS 11CLE2. Fig. 4. Airtime per client comparison for emulation of laptops MG3. showing an intersection of 84.3%, which means that most of the time the operation is similar in both cases. These results indicate that the behavior is quite similar, not only looking at the higher layer considering throughput results, but also taking into account what happens over the air with the actual 802.11-based wireless communication. This fact is important to validate the emulator behavior and ensure that results under different test scenarios would be similar to the ones obtained using laptops for performance testing. C. Emulation results for laptops 11cle2 The validation tests for the laptop model Positivo 11cle2 were conducted with 24 clients, with the emulator operating with the full setup of 3 PCI to mini PCI splitters, each of them with 8 Wi-Fi WNICs. The first analysis corresponds to higher layer performance. In Figure 6 the average TCP throughput per client for both laptops and emulator is shown, while Table I resumes the aggregated averages results. In this case the total aggregated throughput was superior for the emulated clients, with a relative error of 15%. As for the previous laptop model, the variations between laptops throughput is larger than for the emulator clients. In particular, the difference is most noticeable for the laptops which get smaller throughputs, which are clearly below the corresponding emulated clients. When looking at the behavior over the air, the results are summarized in Table IV. The laptops present again a higher variability, which becomes clear with the airtime consumption per client shown in Figure 7, and is consistent with higher layer results. In this case the airtime consumed by retries is almost the same in both cases, while the airtime corresponding to control frames is also quite similar in both tests, with 24% for the laptops and 22% for the emulator. Concerning the packet data rates, Figure 8 shows the two histograms, for both laptops and emulated clients. In this case they present an intersection of 81.7%, a little bit less than for the previous laptop model, but still with a high degree of coincidence. As in the previous case for the MG3 laptops, now for the Positivo 11cle2 model the measurements taken from air captures confirm a similar behavior for both laptops and emulated clients. This fact, in addition to the higher layer results, enable the usage of the emulator as a tool for performance testing, both in the laboratory and in the field. The validation results guarantee that performance evaluation conducted with the emulator will be accurate enough for Plan Ceibal purposes, while avoiding the inconvenience related to the large amount of laptops involved in the tests. Fig. 5. Data rate histogram comparison for emulation of laptops MG3. Fig. 6. Throughput TCP comparison for emulation of laptops 11cle2.

Fig. 7. Fig. 8. Airtime per client comparison for emulation of laptops 11cle2. Data rate histogram comparison for emulation of laptops 11cle2. V. CONCLUSIONS AND FUTURE WORK In this work the design, development and validation of a Wi-Fi multi-client emulation platform was presented. The system developed enables performance testing of high density environments such as classrooms with wireless infrastructure for e-learning. Basically the system is composed by a main computer with several Wi-Fi WNICs and its operation consists of connecting different emulated clients to a wireless network and generating independent traffic from each radio. The prototype was validated for two different laptop models delivered by Plan Ceibal for primary and secondary students. The validation tests were conducted with more than 20 laptops in the 2.4 GHz band and considered both throughput and air capture measurements. This way, several indicators from layer 1 to layer 4 were taken into account, such as throughput, effective data rates and airtime consumption, achieving an acceptable accuracy for all of them. This is a successful first step towards the development of a very useful tool in order to conduct massive performance tests without carrying hundreds of laptops, which contributes to the infrastructure deployment in the education vertical. Among other advantages, the developed system based on a open source Linux system is a low-cost portable solution that can be adapted for emulating specific client devices. This characteristics make it a flexible solution and its modular design allows other users to customize it, for example by changing the WNICs or the corresponding antennas, and also developing their own software for other personalized tests. The user behavior model could also be customized in order to get different traffic generation patterns. New performance metrics for specific applications could be defined and integrated for QoE automatized tests. Other possible uses of the platform developed include performance evaluation of different Wi-Fi WNICs (e.g. purchases for laptops repairing) and also to test Wi-Fi products in order to help the equipment selection process. Regarding future work, the first step would be to enable the operation of multiple emulators simultaneously and perform new validation tests with a higher number of clients. The next step would be to include different possible tests that run under the same platform, considering for example application specific traffic patterns such as online education platforms or video streaming. This is possible because of the modular system design, which also supports future developments that could include directly testing Internet-based applications (e.g. web browsing) avoiding the local server to run the tests. Another line of future work would be the virtualization of the operating system of the emulated clients. This would allow to emulate the user devices more accurately, enabling each virtual machine corresponding to each laptop to run a specific application and generate a specific kind of traffic. This way, different metrics could be obtained to estimate the QoE of each user, concerning the particular application that is running. REFERENCES [1] Extreme Networks, Super Bowl XLVIII Stats Infographic. [Online]. Available: http://www.extremenetworks.com/super-bowl-stats/ [2] International Telecommunication Union, Impact of Broadband on the Economy: Research to Date and Policy Issues, April 2012. [3] Inter-American Development Bank, Development Connections: Unveiling The Impact of New Information Technologies, April 2012. [4] Plan Ceibal, Plan Ceibal: One Laptop per Child implementation in Uruguay. 2007-2015. [Online]. Available: http://www.ceibal.org.uy/ [5] Conectar Igualdad, Argentina, 2010. [Online]. Available: http://www.conectarigualdad.gob.ar/ [6] ConnectED initiative, USA, 2013. [Online]. Available: http://www.whitehouse.gov/issues/education/k-12/connected [7] B. Lewcio and S. Mller, Perception of quality changes in wireless networks, in Quality of Experience. Springer, 2014, pp. 395 410. [8] Iperf, The TCP/UDP Bandwidth Measurement Tool. [Online]. Available: http://iperf.fr/ [9] NS-3, The NS-3 Network Simulator. [Online]. Available: http://www.nsnam.org/releases/ [10] B. A. Chambers, The Grid Roofnet: a Rooftop Ad Hoc Wireless Network, Massachusetts Institute of Technology, 2002. [11] B. Chihani, K. ur Rehman Laghari, E. Bertin, D. Collange, N. Crespi, and T. Falk, User-centric quality of experience measurement, in Mobile Comput., Appl., and Serv. Springer, 2014, vol. 130, pp. 33 46. [12] JP Sá Couto, Laptop Magallanes 3, 2013. [Online]. Available: http://www.jp-ik.com/ [13] Informatica Fueguina SA, Laptop Positivo BGH 11cle2, 2013. [Online]. Available: http://www.positivobgh.com/hogar/spa/home [14] Intel Corp., Centrino Wireless-N 100, 2013. [Online]. Available: http://www.intel.com/content/www/us/en/processors/centrino/centrinowireless-n-100-brief.html [15] Intel Corp., Dual Band Wireless-N 7260, 2013. [Online]. Available: http://www.intel.la/content/www/xl/es/wireless-products/dualband-wireless-n-7260-brief.html [16] Ubuntu, Debian-based Linux operating system. 2015. [Online]. Available: http://www.ubuntu.com/ [17] Metageek, Spectrum analysis with Wi-Spy and Chanalizer. 2015. [Online]. Available: http://www.metageek.com/products/wi-spy/. [18] Wireshark, Network protocol analyzer for Unix and Windows. 2013. [Online]. Available: https://www.wireshark.org/.