Detailed Lab Report Citrix XenDesktop 4 vs. VMware View 4 using Citrix Branch Repeater and Riverbed Steelhead February 11, 2011 Miercom www.miercom.com
Table of Contents 1.0 Executive Summary... 3 2.0 Introduction / Overview... 5 3.0 Test Bed Diagram... 6 4.0 How We Did It... 6 5.1 VDI Quality and Bandwidth, Five Clients, LAN Bandwidth... 8 Figure 1: Average Bandwidth Utilization for VDI Applications with Full LAN Bandwidth... 8 Figure 2: Test 5.1 VDI Client Real-time Bandwidth Consumption with LAN Bandwidth... 9 5.2 VDI Quality and Bandwidth, Five Concurrent Clients, T1 Bandwidth... 10 Figure 3: VMOS Panel Average QoE for VDI Clients with T1 Bandwidth... 11 Figure 4: Bandwidth Consumption for Five VDI Clients with T1... 11 Figure 5: Real-time Bandwidth Consumption for VDI Clients with T1 Bandwidth... 12 5.3 VDI Client Performance with Network Optimization... 13 Figure 6: Test 5.3 VDI Client Real-time Bandwidth Consumption with Riverbed Steelhead Optimization... 13 Figure 7: Test 5.3 VDI Client Real-time Bandwidth Consumption with Citrix Branch Repeater Optimization... 14 Citrix XenDesktop 4 vs. VMware View 4 Page 2 11Feb2011
1.0 Executive Summary Citrix Systems engaged Miercom to validate the user experience and bandwidth consumption of similarly configured desktop virtualization solutions using Citrix XenDesktop 4 and VMware View 4. Since XenDesktop 4 Enterprise and Platinum editions offer many other forms of desktop and application virtualization not available in VMware View 4, these tests focused on the hosted Virtual Desktop Infrastructure (VDI) model in order to enable a fair comparison. Desktop Virtualization through the hosted VDI model uses virtual machines hosted on centrally located servers. Users connect to the virtual machines using a remote display protocol client on their endpoint device. Although users can be geographically scattered, all may be connected to the host servers. Also evaluated in this study was the impact of WAN optimization on the remote display protocols. WAN optimization products are designed to maximize the efficiency of network utilization and performance. This is accomplished through various means including compression, caching and traffic prioritization. The two products used in the study were the Citrix Branch Repeater and the Riverbed Steelhead. Citrix XenDesktop 4 uses HDX technologies to deliver the desktop to the user while VMware View 4 used the PC-over-IP (PCoIP) protocol. This study focuses on these protocols. HDX technologies from Citrix use a TCP-based protocol which is positively impacted by the WAN optimization products which were tested. The PCoIP protocol that VMware uses is UDP-based and was unable to attain an efficiency benefit from the WAN optimization product. The testing methodology was designed to simulate real world user workloads, and evaluate network efficiency (bandwidth) and the overall user Quality of Experience (QOE). User workloads were simulated using an industry standard testing tool LoginVSI from www.loginconsultants.com. This tool uses automated scripts that run tasks using Microsoft Office applications, web browsing, PDFs, and multimedia content. The workloads were tested using a LAN (100 Mbps) for baseline data and on a simulated T1 for WAN conditions. Vendors of competing products were notified in advance and afforded fair opportunity to participate in the testing covered in this report. Vendors were also afforded opportunity to offer technical support to ensure their products were optimally configured. Both VMware and Riverbed declined to actively participate in this test review. The overall findings showed that Citrix XenDesktop 4 was superior in both QoE and network efficiency to VMware View 4 when delivering desktops through hosted VDI. Citrix HDX outperformed VMware PCoIP with and without the benefits of the WAN optimization products. Additionally, the Citrix Branch Repeater significantly improved network efficiency for HDX when compared to the Riverbed Steelhead appliance. Significant Key Findings for tests in this report include: Citrix XenDesktop 4 offers better QoE than VMware View 4 in a five-user concurrent environment for branch deployment (see sections 5.2 and 5.3). Citrix XenDesktop is more bandwidth efficient than VMware View, consuming 43% less bandwidth, making it a better choice for remote branch deployment with limited T1 bandwidth (see section 5.1). Citrix XenDesktop is better optimized using Citrix Branch Repeater than with Riverbed Steelhead (see section 5.5). Citrix XenDesktop 4 vs. VMware View 4 Page 3 11Feb2011
Citrix XenDesktop 4 with Citrix Branch Repeater outperformed VMware View 4 in both Quality of Experience (QoE) and network efficiency for multi-client environment in this side-by-side testing comparison. Miercom is pleased to award the Performance Verified Certification to Citrix for outstanding product performance observed during the testing. Rob Smithers CEO Miercom Citrix XenDesktop 4 vs. VMware View 4 Page 4 11Feb2011
2.0 Introduction / Overview Miercom evaluated the performance of desktop virtualization solutions over branch office networking scenarios, both natively and with WAN optimization technologies. The desktop virtualization solutions tested were Citrix XenDesktop 4 and VMware View 4. The WAN optimization products tested were the Citrix Branch Repeater and the Riverbed Steelhead appliance. The test cases were designed to specifically test bandwidth utilization and Quality of Experience for single and multiple users when accessing Microsoft Office applications which include Word, Excel, Outlook and PowerPoint, as well as Web browsing and Web-based video. The impact of links with restricted bandwidth, as well as network impairments, was measured and recorded. Live video captures were used to graphically show the difference in performance. This document provides observations and results of the tests conducted on XenDesktop 4 and VMware View 4. Products tested were configured in accordance with available documentation and available technical support by the vendor of the product. The display protocols were used out of the box with no optimizations made to either one. Citrix XenDesktop 4 vs. VMware View 4 Page 5 11Feb2011
3.0 Test Bed Diagram 4.0 How We Did It We used an Apposite Linktropy 4500, two Citrix Branch Repeater v5.7.0 optimizers, two Riverbed Steelhead 1050 v6.1.0 optimizers, a VMware server and Citrix XD4 server, a VMware Connection server, a Citrix Desktop Delivery Controller connection server, a Linksys router, a SMC hub, a Netgear GS116 hub, four DELL laptops with Windows 7 and one DELL laptop with Windows XP, five Citrix XenDesktop 4 clients and five VMware View 4 clients running on five laptops. Two HP DL360 servers with two Intel Xeon Quad Core processors were also used for testing. Each server had 12GB of RAM and 3x72GB disk drives for installing the XenDesktop 4 and VMware View 4 suite for VDI testing. We connected two servers to the Apposite WAN emulator through a switch. We used an Apposite WAN emulator www.apposite-tech.com for bandwidth restriction and to introduce packet loss and latency to the network to simulate remote user connections and constricted WAN environments. We also used Login Consultants VSI script www.loginconsultants.com that helped automate the launch of Microsoft applications, including Excel, Word, PowerPoint, Outlook and a video clip in a consistent manner while measuring bandwidth utilization. We measured bandwidth utilization and assessed the QoE while playing Flash movies accessed through Internet Explorer 8 from Web sites including www.hulu.com and www.youtube.com, using the virtual desktop software installed on each client. The tests in this report are intended to be reproducible for customers who wish to recreate them with the appropriate test and measurement equipment. Miercom recommends customers conduct their own needs analysis and testing specifically for the expected environment for the Citrix XenDesktop 4 vs. VMware View 4 Page 6 11Feb2011
product deployment before making a product selection. Contact reviews@miercom.com if you wish to receive assistance from Miercom professional services to conduct these tests. A Video Mean Opinion Score (VMOS) panel was conducted for Test 5.1, 5.2, and 5.3 consisting of 10 individuals. They previewed videos of specific tests which they rated for quality. The scoring system is based on a scale of 0 to 5 0) Fail, 1)Poor, 2) Fair, 3) Good, 4) Very Good, 5) Excellent). The panel was shown pristine versions of the video as a reference point and a worksheet was provided for rating. The practices and procedures of this testing are based on ITU P.800 Standards. The videos sampled are available for review by customers seeking help to make a purchase decision or by vendors who participated in this review, contact reviews@miercom.com Citrix XenDesktop 4 vs. VMware View 4 Page 7 11Feb2011
5.1 VDI Quality and Bandwidth, Five Clients, LAN Bandwidth Objective Establish quality and bandwidth utilization for Citrix XenDesktop 4 and VMware View 4 with five concurrent users without network impairments or the addition of a network optimization appliance and with a 100 Mbps link. Description Each product will be subjected to light, medium and heavy background loads over multiple iterations. Quality of Experience will be captured as well as network resource utilization. The following are the Load Definitions we tested with: Light Load: Microsoft Word, Excel, and Outlook Medium Load: Microsoft Word, Excel, Outlook and PowerPoint Heavy Load: Microsoft Word, Excel, Outlook, PowerPoint, and a video clip Configuration The network environment was unimpaired with 100 Mbps available bandwidth. Observations With a heavy background load over a 100 Mbps LAN link, both XenDesktop and VMware achieved a perfect QoE score of 5.0, with excellent audio and video quality. Average bandwidth utilization for XenDesktop was 43% less than VMware View. See Figure 1 below. The Video Mean Opinion Score (VMOS) Panel used video from this test as a baseline. Figure 1: Average Bandwidth Utilization for VDI Applications with Full LAN Bandwidth Bandwidth (Mbps) 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 (Less is better) 4.1 2.3 Citrix XenDesktop 4.0 VMware Ware View 4.0 Bandwidth consumed when Microsoft Office applications were used for five concurrent clients using VDI Clients on LAN. Citrix XenDesktop 4 vs. VMware View 4 Page 8 11Feb2011
Figure 2: Test 5.1 VDI Client Real-time Bandwidth Consumption with LAN Bandwidth 1 sec 792.4 Kbps 10 sec 1.109 Mbps 1 min 5.934 Mbps 10 min 4.978 Mbps VMware View 4 Bandwidth Consumption 1 sec 113.2 Kbps 10 sec 345.2 Kbps 1 min 1.146 Mbps 10 min 1.959 Mbps Citrix XenDesktop 4 Bandwidth Consumption Running on LAN, Citrix XenDesktop uses about 61% less bandwidth than VMware View without any WAN optimization. Analysis Citrix XenDesktop conserves bandwidth extremely well. In the figure above, we see XenDesktop minimal bandwidth utilization in the absence of any optimization. Citrix XenDesktop 4 vs. VMware View 4 Page 9 11Feb2011
5.2 VDI Quality and Bandwidth, Five Concurrent Clients, T1 Bandwidth Objective Establish baseline performance over a T1 link for Citrix XenDesktop 4 and VMware View 4 with five VDI clients and real-time desktop workload by measuring network resource utilization and Quality of Experience. Description Each product will be subjected to light, medium and heavy background loads over multiple iterations. Configuration WAN emulation constricted to 1.5 Mbps bandwidth with no other impairments. Observation For XenDesktop 4, under light load, QoE achieved a perfect score of 5 with excellent audio and video. Under moderate load, PowerPoint changes were slow and slides with heavy graphics were delayed. Under heavy background load, video was jumpy and audio was choppy. Overall QoE was assessed at 3.0. For VMWare View 4, the View 4 client rendering for basic login was QoE: Fair. The top part of the screen downloaded first, with the remaining screen rendered in stages, not all at once. With moderate load, PowerPoint screens presented artifacting/ghost images and was delayed beyond regular presentation fade effect. With heavy background load, audio was good, but video was choppy, tiled, and did not complete on some clients. Applications with partial screens overlapping each other were observed. Overall QoE was 1.7. See Figure 3 on the following page. In blind panel VMS scoring, XenDesktop achieved an average rating of 4.45 compared to View at 2.35 out of potential score of 5. See Figure 3. XenDesktop consumed 41% less download and 29.91% less upload bandwidth than View with T1. See Figure 4 on the following page. We observed bandwidth utilization was fully consumed at 1.5 Mbps during testing. Citrix XenDesktop 4 vs. VMware View 4 Page 10 11Feb2011
Figure 3: VMOS Panel Average QoE for VDI Clients with T1 Bandwidth VM Ware View 4.0 2.35 Citrix XenDesktop 4.0 4.45 0 2 4 6 Quality of Experience (QoE) QoE rated by blind panel assessment of VDI applications rendering MS Office applications for real-time desktop workload. Scoring: Excellent = 5; Very Good = 4; Good = 3; Fair = 2; Poor = 1; Fail = 0 Figure 4: Bandwidth Consumption for Five VDI Clients with T1 1200 1000 1140 Bandwidth (Kbps) 800 600 400 200 672 0 Citrix XenDesktop 4.0 VMware View 4.0 VMware View uses almost double the bandwidth compared to Citrix XenDesktop on T1. Citrix XenDesktop 4 vs. VMware View 4 Page 11 11Feb2011
Figure 5: Real-time Bandwidth Consumption for VDI Clients with T1 Bandwidth 1 sec 639.3 Kbps 10 sec 1.072 Mbps 1 min 1.211 Mbps 10 min 1.227 Mbps VMware View 4 Bandwidth Consumption 1 sec 66.26 Kbps 10 sec 347.8 Kbps 1 min 670.9 Kbps 10 min 743.0 Kbps Citrix XenDesktop 4 Bandwidth Consumption Running on T1 WAN, Citrix XenDesktop consumed about 40% less bandwidth than VMware View without WAN optimization. Analysis XenDesktop was more economical in its bandwidth utilization and also performed better with regard to user Quality of Experience. In the figure above, we see the ability of XenDesktop to better utilize bandwidth in a T1 constricted environment. The Video MOS Panel scored Citrix XenDesktop 1.89 times higher than VMware View in the constricted T1 environment. XenDesktop provides much better QoE in bandwidth restrained settings. Citrix XenDesktop 4 vs. VMware View 4 Page 12 11Feb2011
5.3 VDI Client Performance with Network Optimization Objective Measure the Bandwidth Consumption of Citrix XenDesktop using MS Office applications over an optimized network with five VDI clients. Description Citrix Branch Repeater and Riverbed Steelhead in separate test scenarios; measure the QoE while Microsoft Office applications are in use over a T1. Configuration Optimization will be provided separately by the Citrix Branch Repeater and the Riverbed Steelhead. Five clients will be running Login VSI scripts with MS Office applications and a video clip over a T1. Figure 6: Test 5.3 VDI Client Real-time Bandwidth Consumption with Riverbed Steelhead Optimization 1 sec 616.0 Kbps 10 sec 1.254 Mbps 1 min 1.081 Mbps 10 min 1.186 Mbps VMware View 4 Bandwidth Consumption 1 sec 456.5 Kbps 10 sec 338.1 Kbps 1 min 355.0 Kbps 10 min 641.1 Kbps Citrix XenDesktop 4 Bandwidth Consumption Running on T1 WAN, with Riverbed Steelhead WAN optimization, XenDesktop consumed about 45% less bandwidth than View. Citrix XenDesktop 4 vs. VMware View 4 Page 13 11Feb2011
Figure 7: Test 5.3 VDI Client Real-time Bandwidth Consumption with Citrix Branch Repeater Optimization 1 sec 1.489 Mbps 10 sec 1.383 Mbps 1 min 1.373 Mbps 10 min 1.260 Mbps Transmission Rate VMware View 4 Bandwidth Consumption 1 sec 1.149 Mbps 10 sec 616.8 Kbps 1 min 253.4 Kbps 10 min 367.9 Kbps Transmission Rate Citrix XenDesktop 4 Bandwidth Consumption With Citrix Branch Repeater, XenDesktop traffic was reduced by over 70% compared to View. This allows either other applications to use the link, or more users to be added without investing in new bandwidth. Table 1: VDI Client with Different Optimization Appliances Branch Repeater Steelhead Bandwidth Saving XenDesktop 323 Kbps 593 Kbps 45% View 1,390 Kbps 1,500 Kbps 7% Bandwidth Saving 77% 61% Bandwidth utilization when testing five clients running Login VSI scripts with different optimization appliances (Citrix Branch Repeater and Riverbed Steelhead) over T1. VMware View consistently used nearly three times the bandwidth compared to XenDesktop on both optimizers. Analysis The bandwidth consumption of Citrix XenDesktop was consistently better than VMWare View when used with either the Citrix Branch Repeater or Riverbed Steelhead. VMware View consistently used three times the bandwidth compared to XenDesktop on the Branch Repeater optimizer (see Table 1 above). As shown in Figures 6 and 7 above, Citrix XenDesktop is better utilizing bandwidth with both optimizers. XenDesktop is best optimized with the Citrix Branch Repeater. XenDesktop is more compressible, using 77% less bandwidth with WAN optimization. Citrix Branch Repeater does the best job of accelerating XenDesktop. PCoIP, the protocol underlying View, cannot benefit from WAN optimization at all. Whether using Citrix Branch Repeater or Riverbed Steelhead, there is only a 7% variation in the results. Citrix XenDesktop 4 vs. VMware View 4 Page 14 11Feb2011