By Ostap Elyashevskyy Automated Performance Testing of Desktop Applications Introduction For the most part, performance testing is associated with Web applications. This area is more or less covered by a lot of articles and videos about how to use JMeter and other tools, which allow load generation and performance metrics capturing. But how do you test the performance of a Desktop application? Long story short, here are some basic requirements: No sources of application (black box) Any Windows application (usually these are resource intensive applications such as arhivators, backup utilities, etc.) Different operations with the application under test and system performance tracking (RAM, CPU, and other metrics) Notification about the system state when the user performs any action (for example, button clicks, message box pop-out (GUI events), etc.) 1
To show how everything works, here is one of the possible approaches to test the performance of Desktop applications. Tool selection UI testing tool In order to capture performance data during user actions, we need to handle the application under test and interact with it, in particular click on controls, emulate mouse and keyboards events, etc. The selected tool should satisfy the following requirements: Interact with any Desktop application: Java,.NET, native Windows app, and others Emulate user actions, clicks, keyboard/mouse events Recognize controls by text/image (language and technology agnostic approach) Capture performance information of actions: when control becomes available, etc. After a short investigation, I have found two possible tools: Sikuli and Al exa. Both are open source projects, with more detailed features description and comparison to be found here. As it s capable of grabbing performance data and integrating with the Nagios monitoring system, Al exa was selected as the main tool for UI performance testing. Resource monitoring tool To analyze metrics like CPU/RAM usage, I/O HDD operations, etc., a resourcemonitoring tool is needed. Basically, Al exa supports integration with Nagios, an open source resource monitoring tool which can be installed on Linux based OS. I have tried it on Ubuntu and found installation/configuration and usage very cumbersome, since it requires installation of NSClient++ on the client side, along with SQL/PHP/ WebServer and other packages on the server side. I needed to get results faster and simplify monitoring tool configuration and deployment, so I used Windows Performance Monitor and integrated it with Al exa. Integration of UI and resource monitoring tool Windows Performance Monitor (WPM) allows custom counters to be created and set though a proxy which will send performance data from Al exa to WPM. Implementation of such a proxy is described later. 2
Solution implementation This section dwells upon implementation of a solution for Desktop Performance Testing by choosing a specific Desktop Application and trying to retrieve metrics. I opted for an installed 7-zip activator that can generate a number of system resources (benchmarking feature). Here s a simple test case for 7-zip: 1. Start 7-zip File manager 2. Click Tools 3. Click Benchmark 4. Wait for 5 seconds 5. Press Stop 6. Close the window The system metrics are measured within the following operations order: a. Navigate menu and open benchmark window b. Perform benchmarking for 10 seconds c. Stop benchmarking 3
Performance Monitor Counters set-up In order to track custom actions, create performance groups and add performance counters for each user workflow. If you are not familiar with Windows Performance Monitor, take a glimpse at this short article describing the basics of WPM. Open Performance Monitor and check an example of Memory counters which are already defined. In addition, there are a lot of other counters related to CPU, HDD, Network, protocols, or specific processes, threads, etc. In this case, add the counters related to a specific test case to see how system performance changes over time during user interactions with application GUI. Having analyzed several solutions about how to set custom counters, I found it s better to do it programmatically on C#. Therefore, I have implemented a small command line utility based on this cool article. Download the files here: C# sources / executable. Let s run utility with these parameters: > CreatePerfCounters.exe ALEXA open_window benchmark stop_benchmark The first parameter sets a group and another one defines counters. By launching 4
this command again, you delete an old group and create a new one with the new counters. Then, open Performance Monitor and check the newly created group with custom counters: The operation of creating counters was separated for two reasons: It s not necessary to create a counter for every test execution This is a resource-consuming operation, so it is faster to create counters separately Proxy implementation Now it s time to set the counters programmatically. To set the counters values, implement proxy in.net (there are probably other solutions implemented on Python, so if you know any libs or better solutions, please share in comments or write me an email). For now, download sources and build them or use a compiled library PerfMonitorProxy.dll. Put this library under the path: C:\PerfMonitorProxy.dll This.NET library will be called by the scripts in Al exa (Python), so it s necessary to re-call it into a.net code. More information about building and calling unmanaged code can be found here and here. For now, it s enough to download and use the library. Getting started with Al exa Tests and interaction with the GUI will be performed by the Al exa tool. First of all, install Al exa. Creating scripts in Al exa IDE is intuitive and the linked getting started manual covers this aspect. Let me elaborate on the most important moments and comment on some code snippets. 5
Performance monitor configuration and capturing results 1. Set up Performance monitor to show only important metrics: 2. Add next counters: 3. Setup grid: Action->Properties->Graph-> Check Vertical and Horizontal grid: 4. (Optional) Click on Data and set Width for custom counters to bold: 5. Click on Memory and set appropriate scaling index (for my computer with 8Gb of RAM, I set 0, 01) This scaling allows Free Memory to 6
be displayed on the same graph, since there is a mark of only 100 on the X axis, zoom down Memory counter. It means that there are ~35 * 100 = 3500 MB free. 6. Try to execute the script and analyze the results and code structure (you may need to change images for controls in the project for your resolution): After script execution, you should be able to see a similar graph: CPU consumption started increasing after we clicked the Benchmark button and dropped down only after we stopped it (blue line). Before we started the test, there was above 3Gb of available memory and after the test it dropped to 2.5Gb During Benchmarking, the highest load of CPU was 87% and just before the stop it had increased to 100%. 7
Now, let s go through the code and check the most essential code snippets: proxy.perfmon( ALEXA, open_window, 50) This command executes functions from our proxy dll library and sets the performance counter open_window with value 50 (you can set any value as it is used to visualize the start of the action). I am calling the end action: proxy.perfmon( ALEXA, <any counter>, 0) to set the counter s value to 0. In case you need to wait until the dynamic controls appear, use Bind function: performancevalue = ControlToWait.Bind(15) # 15 seconds - wait timeout 8
After that, send its value to output. Here s how it may look like: OK: Open Tools time is 0.842000007629 sec. Recording performance data to log file The previous chapter described capturing and showing results in real time mode. Usually this is needed to schedule tests and collect results to analyze it afterwards. Here s how to record performance test data in the file: 1. Right click on Data Collection Sets -> User Defined->New->Data Collector Set 2. Name it: AlexaMetrics, click Next 3. Chose Basic, click Next 4. Choose location for logs or leave as is and click Next 5. Click Finish 6. Now add counters: 9
7. Click start to capture metrics 8. Launch the script 9. Stop recording performance data Now let s check what has been recorded: 1. Click on Performance Monitor 2. Chose properties and choose log file you have captured: 10
3. After clicking Apply, you should see something similar to this: Summary Performance Windows Monitor enables tracking of all the important metrics and system/process parameters. In addition, it can capture performance metrics to CSV, log, binary, tabulated files, or SQL. Also WPM provides the opportunity to schedule monitoring execution, define stop conditions, execute all types of actions on specific conditions, etc. The combination of Al exa and WPM results in a powerful flexible test and monitoring system for Windows desktop applications. When tracking applications from different hosts, opt for integration with Nagios or other monitoring systems. Scripts and code provided in this article were created as a proof of concept and can be improved and applied according to your needs. 11
About Author Ostap Elyashevskyy is a Test Automation Expert at SoftServe. He has more than 7 years of experience, 2 of which are in software development and 5 are in test automation testing. The main projects Ostap worked on are mobile test automation projects for both Android and ios platforms, Dekstop and Web projects where various types of testing were implemented - from functional to performance/load and static analysis. Ostap believes that automation should be applied not only for software testing, but also other activities where time can be saved and automation brings benefits: collecting metrics of different kinds, implementation of helpful utilities such as bug migration utility or test results synchronization with test management system, etc. About SoftServe SoftServe is a leading technology solutions company specializing in software development and consultancy services. Since 1993 we ve been partnering with organizations from start-ups to large enterprises to help them accelerate growth and innovation, transform operational efficiency, and deliver new products to market. To achieve this we ve built a strong team of the brightest, most inquiring minds in the industry, and we form close, collaborative relationships with our clients so we can really understand their needs and deliver intuitive software that exceeds their expectations. Our experience stretches from Cloud, Security and UX Design to Big Analytics and the Internet of Things, we have offices across the globe and development centers across Eastern Europe. For more information please visit www.softserveinc.com USA HQ Toll Free: 866-687-3588 Tel: +1-512-516-8880 Ukraine HQ Tel: +380-32-240-9090 Bulgaria Tel: +359-2-902-3760 Germany Tel: +49-69-2602-5857 Netherlands Tel: +31-20-262-33-23 Poland Tel: +48-71-382-2800 Sweden Tel: +46-844-685-084 UK Tel: +44-207-544-8414 EMAIL info@softserveinc.com WEBSITE: www.softserveinc.com