Performance Testing in Virtualized Environments Emily Apsey Product Engineer
Introduction Product Engineer on the Performance Engineering Team Overview of team - Specialty in Virtualization - Citrix, VMWare, Microsoft Dedicated to Product Virtualization - Configuration - Scalability/Density - Provide feedback loop across all development teams
Collaborative Testing Vendors - Active participation by vendors - Hands on configuration and tuning - NVIDIA - Hardware - SuperMicro and Dell r720 - K1 and K2 cards - Driver (on hypervisor and VM) Microsoft - Issaquah labs - Nov 13 - RemoteFX team VMWare - EEC lab Nov 14 - vsga Jan 14 - vgpu Oct 14 - Future Reference Architecture Doc Citrix - Majority of our Users
Presentation Overview Why and how we tested using Nvidia GRID cards - Test methodology for virtualized test environments Lesson Learned Future Testing
Virtualization Esri ArcGIS in virtualized environments - Users began virtualizing our ArcMap software - Improve understanding of performance and scalability of ArcGIS in virtualized environments - Improve information, internally and externally, related to: - Virtualization environments - What to expect - Performance - Scalability - New technologies - NVIDIA K1 and K2 cards
What and Why? ArcGIS Pro - New addition to ArcGIS Desktop suite - Desktop- analysis, publication, visualization - New rendering engine - DirectX11 and OpenGL - Currently most heavily used app - ArcMap, based on GDI+ - Heavily used in Citrix XenApp - Must virtualize well - Provide great UX
Testing Goals Show if a shareable GPU is needed (ArcMap + Pro) - ArcMap single threaded, GDI+ rendering - Pro multi threaded, DX or OpenGL rendering GPU needed for both 2D and 3D workflows? - How to monitor and communicate performance/ scalability Optimal VM configurations - vcpu, Memory, GPU Profile for optimal user experience - What and how to communicate information to users, IT administrators - How many VMs can a single GRID card accommodate? Administration - Know a little before you go on-site or create a demo of ArcGIS Desktop in virtualized environments
Graphics Options in Virtualization
GRID vgpu Profiles
GPU Profile Citrix and VMWare vgpu - Amount of Frame Buffer - Application requires at least 1 GB of FB - Identify Optimal profile for both VMWare and Citrix Hypervisors - Dependent on data and workflow - Effects density Microsoft Hyper-V - Does not use GPU Profiles - Some control by screen resolution
Data and Workflows 2D Denver - Local government Vector data model - Database- Points, line and polygons 3D Virtual City - Philly Raster data, Z values extruded, building textures Zoom/Pan to a series of standardized bookmarks within App - Time rendering (Sum of all bookmarks) - Frames Per Second (FPS) - Average Sample every 100 milliseconds - Minimum Indicates jitter, poor user experience
Testing Tools Visual Studio vs. Light-weight application. - VS -performance overhead on each VM: disk, startup, GPU - UI maps maintenance overhead Login VSI - Trail period, can be scriptable Accurate Performance Metrics within Application - Internally developed ArcGIS Pro Add-In - Exposure of valuable metrics - Propagating events into an API method. - Very accurate performance metrics - Used by developers - Light weight
ArcGIS Pro Add-In Specifically designed for performance Script and reproducible Self contained Packaging data and test package to business partners Great for Demos!
ArcGIS Pro Add-In UI
Automate Test Execution Test Controller - Tests are executed via command line - Calls upon a python script which creates windows scheduled tasks - Opens ArcGIS Pro application and project - Rendering events logged - Automatically closes ArcGIS Pro when test completes Scalability across multiple VM s 1, 4, 8, 12, 16 VM s - TestList.csv - Test Cleanup - Collect logs from VM s to File Share or SAN - Unique Names VM Name
Collecting Data ArcGIS ProAddIn Log - Draw Times, FPS Avg and Min - Named Metrics - Licensing Time - Open Project Time ArcMon (ArcGIS Pro) Internal Use only - Hung Threads Nvidia SMI - GPU Utilization, GPU Memory Utilization ESXiTop (VMWare) - CPU Utilization (Host and VM) Python based parsers - Export to log which is useable for reporting purposes - Drawback - No automated UI to show system metrics
ArcGIS Pro Add-In Log Python based parser 2D SumBkAvg StDev FPSAvg FPSMin 1 92.51 66.65 42.06 4 87.78 4.44 66.60 42.93 8 100.62 6.75 65.99 37.54 12 106.86 4.13 69.30 37.54 16 111.95 4.95 74.11 23.70 3D SumBkAvg StDev FPSAvg FPSMin 1 81.00 54.02 13.74 4 79.66 4.35 53.41 14.65 8 86.95 4.33 47.38 10.71 12 92.00 6.19 43.23 9.64 16 103.42 8.00 40.91 7.66
Additional Hands-On Testing Further validate automated test findings Usability tests were executed manually by team members Further qualifying the UX - Validating rendering quality to human eye - Overall responsiveness of the application User Experience Ranking 1 to 5 Unacceptably jerky; Poor to annoying user experience 6 Jerky, but useable 7 Moderately smooth, moderate jerkiness 8 Smooth, minor jerkiness 9 to 10 Very smooth; little to no difference between a non-vdi solution. No apparent jerkiness, lag/tearing, or texturing delays.
100% Host CPU Utilization Scalability of 240Q Profile - 3D Data 90% 80% 70% 60% 50% 40% 30% 16 VMs:Host CPU 14 VMs:Host CPU 12 VMs:Host CPU 10 VMs:Host CPU 20% 10% 0%
16:13:02 16:13:18 16:13:33 16:13:49 16:14:04 16:14:20 16:14:35 16:14:51 16:15:07 16:15:22 16:15:38 16:15:54 16:16:09 16:16:25 16:16:40 16:16:56 16:17:11 16:17:27 16:17:43 16:17:58 16:18:14 16:18:29 16:18:45 16:19:01 16:19:16 16:19:32 16:19:47 16:20:03 16:20:18 16:20:34 16:20:50 16:21:05 16:21:21 16:21:36 16:21:52 16:22:08 16:22:23 16:22:39 16:22:54 16:23:10 16:23:25 16:23:41 100 Host Utilization Rates 16VMs - 6vCPU: 3D data 90 80 70 60 50 40 GPU1 %Util GPU2 %Util GPU3 %Util GPU4 %Util Cpu(Total)\% Util Time 30 20 10 0
Answers obtained from testing ArcGIS Pro - GPU is required for 3D data; K2 card preferred - GPU aids 2D data delivery by relieving CPU; considered recommended. GPU Utilization GPU can be configured to be shareable (vgpu) - Exception XenApp; requires Pass Through GPU - Optimal VM configuration can vary by HyperVisor (encoding/decoding)
ArcGIS Pro XenApp 7.6 100 90 80 70 60 50 40 30 20 XenApp 7.6-4 Users 3D 16vCPU, 16 GB of Memory Requires GPU Pass-Through - K2 GPU Memory = 4GB OS does not do well splitting GPU resources across different terminal sessions XenDesktop recommended - Dedicated resources - vgpu profile manages FB - Dictates VM density Esri collaborating with Nvidia and Citrix 10 0 XenApp76\% Processor Time XenApp76\% GPU Usage XenApp76\% GPU Memory Usage
Recommended GPU Profiles Dependent on user data and workflows 3D Data 2D Data 2D Data
Answers obtained from testing (continued) ArcMap Rendering engine does not utilize GPU - GPU is not heavily utilized - Only when user explicitly executes hardware acceleration operations Map docs Rendering Sum Times (mins)
Lesson Learned Online data introducing network latency - Eliminate ArcGIS Online data for initial benchmarks Licensing - License Server vs. Application licensing - Eliminates logins into Application FPS sampling Avg and Min Think time (5 seconds vs 10 seconds) - Aggressively intensive, further refine to closely model user workflows Nvidia Drivers HyperVisor and VM, same release
Future testing Mixed workflows - Initial testing focused on rendering pipeline - Add analytical operations (CPU intensive) - How GPU Utilization and density is affected when users are simultaneously running a CPU intensive geoprocessing task? Mixed Virtualization Vendor Solutions - VMWare ESXi XenDesktop Storage SAN - More realistic large infrastructure don t use SSDs Cloud based VDI (DaaS) - Increasing questions and interest - Distributors Europe, NZ, US - Esri Redlands Ed Services, Customer Holistic Testing
Questions? Emily Apsey - eapsey@esri.com