Performance Testing in Virtualized Environments. Emily Apsey Product Engineer

Transcription

1 Performance Testing in Virtualized Environments Emily Apsey Product Engineer

2 Introduction Product Engineer on the Performance Engineering Team Overview of team - Specialty in Virtualization - Citrix, VMWare, Microsoft Dedicated to Product Virtualization - Configuration - Scalability/Density - Provide feedback loop across all development teams

3 Collaborative Testing Vendors - Active participation by vendors - Hands on configuration and tuning - NVIDIA - Hardware - SuperMicro and Dell r720 - K1 and K2 cards - Driver (on hypervisor and VM) Microsoft - Issaquah labs - Nov 13 - RemoteFX team VMWare - EEC lab Nov 14 - vsga Jan 14 - vgpu Oct 14 - Future Reference Architecture Doc Citrix - Majority of our Users

4 Presentation Overview Why and how we tested using Nvidia GRID cards - Test methodology for virtualized test environments Lesson Learned Future Testing

5 Virtualization Esri ArcGIS in virtualized environments - Users began virtualizing our ArcMap software - Improve understanding of performance and scalability of ArcGIS in virtualized environments - Improve information, internally and externally, related to: - Virtualization environments - What to expect - Performance - Scalability - New technologies - NVIDIA K1 and K2 cards

6 What and Why? ArcGIS Pro - New addition to ArcGIS Desktop suite - Desktop- analysis, publication, visualization - New rendering engine - DirectX11 and OpenGL - Currently most heavily used app - ArcMap, based on GDI+ - Heavily used in Citrix XenApp - Must virtualize well - Provide great UX

7 Testing Goals Show if a shareable GPU is needed (ArcMap + Pro) - ArcMap single threaded, GDI+ rendering - Pro multi threaded, DX or OpenGL rendering GPU needed for both 2D and 3D workflows? - How to monitor and communicate performance/ scalability Optimal VM configurations - vcpu, Memory, GPU Profile for optimal user experience - What and how to communicate information to users, IT administrators - How many VMs can a single GRID card accommodate? Administration - Know a little before you go on-site or create a demo of ArcGIS Desktop in virtualized environments

8 Graphics Options in Virtualization

9 GRID vgpu Profiles

10 GPU Profile Citrix and VMWare vgpu - Amount of Frame Buffer - Application requires at least 1 GB of FB - Identify Optimal profile for both VMWare and Citrix Hypervisors - Dependent on data and workflow - Effects density Microsoft Hyper-V - Does not use GPU Profiles - Some control by screen resolution

11 Data and Workflows 2D Denver - Local government Vector data model - Database- Points, line and polygons 3D Virtual City - Philly Raster data, Z values extruded, building textures Zoom/Pan to a series of standardized bookmarks within App - Time rendering (Sum of all bookmarks) - Frames Per Second (FPS) - Average Sample every 100 milliseconds - Minimum Indicates jitter, poor user experience

12 Testing Tools Visual Studio vs. Light-weight application. - VS -performance overhead on each VM: disk, startup, GPU - UI maps maintenance overhead Login VSI - Trail period, can be scriptable Accurate Performance Metrics within Application - Internally developed ArcGIS Pro Add-In - Exposure of valuable metrics - Propagating events into an API method. - Very accurate performance metrics - Used by developers - Light weight

13 ArcGIS Pro Add-In Specifically designed for performance Script and reproducible Self contained Packaging data and test package to business partners Great for Demos!

14 ArcGIS Pro Add-In UI

15 Automate Test Execution Test Controller - Tests are executed via command line - Calls upon a python script which creates windows scheduled tasks - Opens ArcGIS Pro application and project - Rendering events logged - Automatically closes ArcGIS Pro when test completes Scalability across multiple VM s 1, 4, 8, 12, 16 VM s - TestList.csv - Test Cleanup - Collect logs from VM s to File Share or SAN - Unique Names VM Name

16 Collecting Data ArcGIS ProAddIn Log - Draw Times, FPS Avg and Min - Named Metrics - Licensing Time - Open Project Time ArcMon (ArcGIS Pro) Internal Use only - Hung Threads Nvidia SMI - GPU Utilization, GPU Memory Utilization ESXiTop (VMWare) - CPU Utilization (Host and VM) Python based parsers - Export to log which is useable for reporting purposes - Drawback - No automated UI to show system metrics

17 ArcGIS Pro Add-In Log Python based parser 2D SumBkAvg StDev FPSAvg FPSMin D SumBkAvg StDev FPSAvg FPSMin

18 Additional Hands-On Testing Further validate automated test findings Usability tests were executed manually by team members Further qualifying the UX - Validating rendering quality to human eye - Overall responsiveness of the application User Experience Ranking 1 to 5 Unacceptably jerky; Poor to annoying user experience 6 Jerky, but useable 7 Moderately smooth, moderate jerkiness 8 Smooth, minor jerkiness 9 to 10 Very smooth; little to no difference between a non-vdi solution. No apparent jerkiness, lag/tearing, or texturing delays.

19 100% Host CPU Utilization Scalability of 240Q Profile - 3D Data 90% 80% 70% 60% 50% 40% 30% 16 VMs:Host CPU 14 VMs:Host CPU 12 VMs:Host CPU 10 VMs:Host CPU 20% 10% 0%

20 16:13:02 16:13:18 16:13:33 16:13:49 16:14:04 16:14:20 16:14:35 16:14:51 16:15:07 16:15:22 16:15:38 16:15:54 16:16:09 16:16:25 16:16:40 16:16:56 16:17:11 16:17:27 16:17:43 16:17:58 16:18:14 16:18:29 16:18:45 16:19:01 16:19:16 16:19:32 16:19:47 16:20:03 16:20:18 16:20:34 16:20:50 16:21:05 16:21:21 16:21:36 16:21:52 16:22:08 16:22:23 16:22:39 16:22:54 16:23:10 16:23:25 16:23: Host Utilization Rates 16VMs - 6vCPU: 3D data GPU1 %Util GPU2 %Util GPU3 %Util GPU4 %Util Cpu(Total)\% Util Time

21 Answers obtained from testing ArcGIS Pro - GPU is required for 3D data; K2 card preferred - GPU aids 2D data delivery by relieving CPU; considered recommended. GPU Utilization GPU can be configured to be shareable (vgpu) - Exception XenApp; requires Pass Through GPU - Optimal VM configuration can vary by HyperVisor (encoding/decoding)

22 ArcGIS Pro XenApp XenApp Users 3D 16vCPU, 16 GB of Memory Requires GPU Pass-Through - K2 GPU Memory = 4GB OS does not do well splitting GPU resources across different terminal sessions XenDesktop recommended - Dedicated resources - vgpu profile manages FB - Dictates VM density Esri collaborating with Nvidia and Citrix 10 0 XenApp76\% Processor Time XenApp76\% GPU Usage XenApp76\% GPU Memory Usage

23 Recommended GPU Profiles Dependent on user data and workflows 3D Data 2D Data 2D Data

24 Answers obtained from testing (continued) ArcMap Rendering engine does not utilize GPU - GPU is not heavily utilized - Only when user explicitly executes hardware acceleration operations Map docs Rendering Sum Times (mins)

25 Lesson Learned Online data introducing network latency - Eliminate ArcGIS Online data for initial benchmarks Licensing - License Server vs. Application licensing - Eliminates logins into Application FPS sampling Avg and Min Think time (5 seconds vs 10 seconds) - Aggressively intensive, further refine to closely model user workflows Nvidia Drivers HyperVisor and VM, same release

26 Future testing Mixed workflows - Initial testing focused on rendering pipeline - Add analytical operations (CPU intensive) - How GPU Utilization and density is affected when users are simultaneously running a CPU intensive geoprocessing task? Mixed Virtualization Vendor Solutions - VMWare ESXi XenDesktop Storage SAN - More realistic large infrastructure don t use SSDs Cloud based VDI (DaaS) - Increasing questions and interest - Distributors Europe, NZ, US - Esri Redlands Ed Services, Customer Holistic Testing

27 Questions? Emily Apsey - eapsey@esri.com