Selecting Automated Testing Tools

Ensuring Software Success SM Selecting Automated Testing Tools A SmartBear White Paper Selecting the right automated software testing tool involves having a list of requirements to review during your evaluation. Without a list of requirements, you may waste time downloading, installing and evaluating tools that only meet some of your requirements. This White Paper outlines common requirements for automated testing tools which you can build from to create a list that reflects the specifics of your software testing needs. Contents Introduction... 2 Support for Various Applications and Platforms... 2 Support for Various Operating Systems... 3 Support for Various Testing Types... 3 Creating Automated Tests Without Programming... 4 Automated Test Scripting... 5 Recording Automated Software Tests... 5 Creating Automated Tests that are Resistant to Changes... 6 Support for Controls and Objects... 6 Support for Various Test Data Sources... 7 Running Automated Software Tests and Synchronization...7 Automatically Compare Results of Automated Test Actions...8 Logging Automated Test Results...8 Exporting Reports...8 Additional Needed Automated Testing Tool Features...9 Testing Multi-Language Applications...9 Automated Testing Tool: Support for Team Work...9 Command Line and OLE Automation Support... 10 Integration with Team System and Build Software... 10 Technical Support... 10 Pricing Policy... 10 Free Trial Version... 11 www.smartbear.com/testcomplete

Introduction Testing is a critical part of the software development process. There are a lot of different automated software testing tools currently on the market. Some of these tools are only able to perform specific kinds of testing and only work with specific languages, like, for example, Java application unit testing. Other products support a wide range of applications and offer more features and functionality. When you start your search for the right automated software testing tool, it s important to create a list of requirements to review when choosing a tool for evaluation. If you don t have a list of requirements, you may waste time downloading, installing and evaluating tools that only meet some of your requirements, or may not meet any of them. Creating a list of requirements is not easy, since you have to take all of your testing needs into account. We have prepared a list of requirements, which are described below. Of course, the list does not include every situation, but this list is a good start. You can modify and add to it to get a list that reflects the specifics of your software testing environment. Support for Various Applications and Platforms Some automated testing tools only support one type of application, like, for instance, only Java or.net applications, while other tools support more application types. In general, the best option is to choose a tool that supports all development tools currently used in your organization, or all tools that you plan on using in the near future. Even if you build your application with one compiler, you might use other compilers in the future. Selecting one multi-compiler software testing tool instead of several compiler-specific tools may save you money, but you must decide which is more important and whether it will save you time and energy when creating tests for future application versions. Purchasing and using several software testing tools requires additional time for training and creating tests. Do you have that much time? Does your company have enough budgeted to pay for this? TestComplete is one of the automated software testing tools that support the testing of any Windows program created with any compiler. It works with both 32-bit and 64-bit Windows,.NET, Java and web applications. It can also test applications running on Windows Mobile-based PDAs. TestComplete s automated test engine provides extended support for the following applications: C#, Visual Basic.NET, VCL.NET and any other.net application for any version of.net Framework (1.0, 1.1, 2.0, 3.0, 3.5 or 4.0) WPF (XAML) and XBAP applications (.NET Framework 4) Java applications (including Swing, SWT, AWT and WFC applications) Visual C++ 6.0-9.0 applications (including ATL, MFC and WTL applications) Visual Basic 6.0 Delphi and C++Builder applications (both VCL and CLX applications) 2

Flex, Flash, Silverlight and JavaFX applications Windows Mobile applications Qt applications wxwidgets applications This extended support means that TestComplete can access the internal objects, methods and properties of the application under test. QA engineers can use all of these elements in their automated tests. For instance, your tests can execute an object s internal methods to initiate a unit test. TestComplete also provides access to methods and properties of web page elements displayed in Internet Explorer, Firefox, Netscape Navigator or any browser based on the Microsoft Web Browser control. It can test web pages on the client computer and is not dependant on the server technology. You can use it to test web pages generated by IIS, ASP, ASP.NET, PHP, CGI, AJAX, Atlas and other applications. Support for Various Operating Systems Choose an automated testing tool that supports as many operating systems as possible. Make sure that the tool supports recent operating system versions. The fact that your customers do not run your application under Windows Vista does not mean that they will not use it in the future. If your application works under this operating system, your software testing tool should be able to work under it as well. In other words, the more Windows versions the software testing tool supports, the better. TestComplete supports both old and new Windows operating systems: Windows 2000, Windows XP, Windows Vista, Windows 7, Windows Server 2003 and Windows Server 2008, including both 32 and 64-bit versions of these operating systems. TestComplete also provides automated testing support for Windows Mobile applications running on PDAs. The range of supported operating systems on PDAs includes Windows Mobile 2003 (Pocket PC 2003), Windows Mobile 2003 SE (Pocket PC 2003 SE), Windows Mobile 5.0 and Windows Mobile 6.0. Support for Various Software Testing Types When choosing an automated testing tool, check whether it supports the test types that you are going to run. Can it simulate user actions over the tested application s user interface, in other words, perform functional testing? Can it run unit tests built into the tested application? Can it run tests created with NUnit, JUnit, DUnit or MSTest? Does it support distributed testing? Does it have special features for synchronizing tests that run simultaneously on several network workstations? Does it support load testing? Also, if you test your products using both manual and automated methods, you may want to choose a tool that provides specific support for manual tests. Also, don t forget other important factors, such as, data-driven testing support and the ability to use the tool for regression testing. There are a lot of automated software testing tools that support only one or two testing types, for instance, only unit testing or functional testing. Some vendors require separate products for different testing types. For instance, support for load or distributed testing can be implemented with separate products or add-ons. If you wish to support these testing types, you will have to add the price of these products or add-ons to the price that you are going to pay for the product. 3

TestComplete supports unit, functional (or UI), regression, data-driven, manual and distributed testing of Windows applications. It provides special support for unit tests created in NUnit, JUnit, DUnit, MSTest, which allows you to run these tests directly from your TestComplete projects. This lets you use unit tests that were created with these tools earlier in your development cycle. With TestComplete you can also perform load, stress and scalability testing of Web servers and services. To perform coverage testing, check the application for resource and memory usage and you can find bottlenecks in your application by using another one of SmartBear s products called AQtime. AQtime is tightly integrated with TestComplete, so you can run coverage, performance and memory leak tests directly from your TestComplete projects. Creating Automated Tests Without Programming An automated testing tool must let you create tests fast and effectively. The test creation techniques must be sufficiently powerful to allow performing various automated testing tasks. At the same time the automated testing tool must be easy-to-use and clear even for inexperienced QA personnel. A good automated testing tool may support several ways for creating automated tests. For instance, it may provide a simple, easy-to-use visual automated test editor and some other features, like automated test scripts, for performing advanced tasks. Many automated testing tools only provide scripting capabilities for creating automated tests. Creating automated test scripts require programming experience and may be difficult for inexperienced testers. TestComplete provides a simple but powerful alternative to automated test scripts: keyword tests (also called keyword-driven testing). Keyword tests consist of operations that simulate user activities on the application under test (like clicking the mouse buttons or typing on the keyboard) or perform other automated testing actions: launching tested applications, posting messages to the test log, running other automated tests (for instance, low-level procedures or unit tests), use checkpoints to verify specific application functionality and more. Keyword tests are very simple and easy to create. Testers do not need programming skills to create them. You can record these keyword-driven tests or modify them visually in a special editor. The editor s intuitive UI, wizards and dialogs help even inexperienced users create powerful automated tests with ease. Keyword tests are an ideal feature for creating functional tests quickly. But keyword tests are not the only automated tests that can be recorded or created visually. TestComplete can record automated test scripts, load tests and low-level procedures (special procedures that simulate low-level events like pressing and releasing of mouse buttons or keyboard keys). Load tests and low-level procedures can then be modified visually with special editors. Recorded automated test scripts can also be modified in the code editor. 4

Automated Test Scripting Many automated software testing tools also provide scripting capabilities. Scripts are very important when performing automated testing and they let users create automated tests for specific application features, provide a variety of automated testing actions and perform those operations that cannot be done with standard features built into the automated testing tool. Automated test scripts are useful even if the automated testing tool provides easier ways to create automated tests, because script code typically provides more powerful functionality, it lets you perform those actions that other automated testing types cannot. If you prefer scripting for your automated testing, it is important to pay attention to the scripting language the automated testing tool uses. Using standard languages, such as VBScript or JScript rather than their dialects, will decrease the time needed to get acquainted with a product. Another benefit is that you can find information on these languages from third-party web sites as well as MSDN. Two other important features to think about when choosing a tool that uses scripting is the code editor and script debugger. Check whether the editor supports features that simplify and speed up script writing: syntax highlighting, outlining, code completion, bookmarks, Ctrl-click navigation between script routines and many others. Check if the tool s script debugger includes the needed debugging functionalities, such as breakpoints, step-over and step-into execution, Call Stack window and Evaluate dialog, etc. Using all of these features you can easily find and correct bugs or find workarounds. TestComplete supports several scripting languages: VBScript, JScript, DelphiScript, C++Script and C#Script, so, you can choose the language you know best and thus save more time. TestComplete includes a powerful code editor that supports all modern code editor features: syntax highlighting, outlining, code completion, code templates, bookmarks and many others. TestComplete s built-in script debugger supports ordinary and conditional breakpoints, step-over, step-into and run-to-the-cursor features, the Watch and Call Stack windows and special features for evaluating variables and objects during the script run. Recording Automated Software Tests Functional (or user interface) testing implies simulating user actions on the application under test. Implementing these tests even for a small application is a tiresome process. When choosing an automated testing tool, make sure the tool can record and play back user actions on the tested application. This feature will significantly streamline test development. TestComplete records user actions as keyword tests or automated test scripts, which you can then edit and play back an unlimited number of times. Fast test development is not the only benefit. Recording also helps you understand how to create a keyword test or how to write automated test script code, how to address application objects and how to perform testing actions on them. We d like to note that TestComplete not only records and plays back tests; it also records and plays back HTTP traffic. This feature lets you easily create load tests for web servers and services. TestComplete can also filter out traffic sent and received by specific applications during traffic recording. 5

Creating Automated Tests that are Resistant to Changes Normally, you start testing your application while it is still in development. So, it is quite possible that after you make changes to the application, the current tests will become obsolete and you ll have to update the test to work with the changed application windows and controls. Modifying the tests may take the same amount of time as modifying the application itself. That s why smart teams like to create flexible tests that will be resistant to the changes made to the application under test. One approach that decreases the dependence on changes made to the tested application is the object-oriented testing model. Each window or control of the application under test is represented by an appropriate object with the name, for instance, [WinFormsObject( Form1 )], so, to simulate user actions on this window or control, you call the object s methods and properties. For instance, simulating a mouse click on a button can be represented as a call to the Click method of the test object that corresponds to this button. If you use this approach, there is no need to change your test after the tested application has changed. For instance, if developers change a button s width, height or position, the Click method will still work. If you simulate a click by coordinates, changing the button s size or position would cause changes in the automated test. Nevertheless, in some cases, you may need to work with coordinates as well, so, it is nice if the automated testing tool supports both object-oriented and coordinate-oriented methods. Another approach that lets you create robust tests is object mapping. As a rule, automated software testing tools address applications objects by their names or captions. However, during application development, programmers may change controls captions, so you will have to modify your tests in order for them to match the changed controls. That is why a good automated testing tool should be able to map an object name to a custom name and use this custom name to address the object in tests. However, the tool must be able to map an object not only by its caption or identifier. In this case, if a developer changes the tested application, you will only have to modify the mapping conditions while the scripts and other tests remain the same. TestComplete includes special features that let you map application windows, controls and objects by their properties. Testers can map a short name to the tested object and then use this mapped name in automated tests. This simplifies working with objects, makes the created tests shorter and easier to understand. There is another benefit of name mapping - it makes automated tests independent of object properties, which may change from one build to another. Also, TestComplete supports the object-oriented approach to simulate user actions. Simulating mouse events by coordinates is also supported. Support for Controls and Objects Windows applications use controls to display information and interact with users. The operating system provides dozens of various controls that can be used in tested applications: edit boxes, list views, check boxes, menus, buttons and much more. This number is also extended with custom components offered by third-party vendors. These can be non-standard components, like a tree list or grid, or third-party analogs to standard components: custom menus, toolbars and many others. When choosing an automated testing tool, make sure it includes special features that simplify the work with controls used in your application. It is important to determine whether 6

these features are for both standard and third-party controls. Another important factor is access to methods and properties of the application s internal objects and controls. This access provides methods and properties for simulating user actions and checking the results of test actions. For example, you can use the internal methods and properties of a grid control to change the value in a grid s cell and then check if the change was successful. TestComplete provides specific methods and properties that greatly simplify your work with standard Win32 controls, web page controls, WPF controls and third-party components provided by Microsoft, Borland, ComponentOne, Developer Express, Infragistics, Syncfusion, Telerik and other vendors. In addition, TestComplete s automated test engine can access the internal methods and properties of application objects and use these methods and properties to perform the desired testing tasks. Supports Various Test Data Sources When performing functional testing, it s a common practice to simulate user input into the tested application s data forms. Thorough testing implies that you feed varied data to the input fields and that the tested application processes it correctly. When choosing an automated testing tool, check which data formats it can use, such as text files, XML files, database tables, and others. With TestComplete, you can obtain data from text and XML files, Excel worksheets and databases like SQL Server, Oracle and MySQL. TestComplete includes special program objects and test operations that allow you to iterate through data records in external files and easily create data loops in your automated tests. You can create and populate these data storages manually, or you can use TestComplete s Data Generator to generate the desired test data automatically and save it either to TestComplete s Table variable or to the specified Excel file. This approach helps you decrease the time spent on preparing test data. For more information on using this feature, see the Using Data Generators section in TestComplete s help. TestComplete also allows you to use script arrays, variables and constants to keep data for automated tests. Running Automated Software Tests and Synchronization The state of the application under test and the execution environment may differ from the state and conditions you expected. For instance, some network resource may be unavailable, or the desired window is not created; the web page loads very slowly or an unexpected modal dialog box appears. These and other similar factors may cause the automated software testing tool to decide that the test has failed and it should be stopped. A good automated testing tool must be able to handle these situations in the appropriate manner. For instance, it must be able to wait for the needed windows, close unexpected dialogs automatically and perform other actions that a human would do to continue the test. TestComplete can handle these situations properly. For example, methods that work with windows and web pages can wait until a window appears on screen or until a web page is loaded; methods that work with processes can wait until the desired process starts and so on. TestComplete also includes special features that let you 7

specify how the tool should handle unexpected windows without interrupting the test run. Automatically Compare Results of Automated Test Actions An automated testing tool should include features for automatically comparing the results of test actions. For instance, if during the test run the tested application saves a customer list to a file, the automated testing tool should be able to compare this file with a baseline copy and determine whether the save was successful and the saved data is correct. The automated testing tool should be able to compare objects and controls properties, images, files and other objects. TestComplete includes special tools, objects and functionality that you can use to compare files, images and object properties. The comparison can be implemented manually, or generated by TestComplete when you are inserting a checkpoint in your test. TestComplete includes special dialogs and wizards that can be used to create checkpoints for properties, objects, files, images, databases, tabular controls, XML documents, web services and web pages. By using the powerful test implementation tools, you can create automated tests that will compare complex objects and can also use regular expressions for comparisons. Logging Automated Test Results An automated software testing tool should keep a log of all testing actions and provide a detailed report for them. The log of a high-end software testing tool must contain different types of messages: errors, warnings and informative messages. In this case, you can easily determine what went wrong and why the test failed. TestComplete includes a powerful test log that contains warning, error or informative messages as well as files and images. The images are captured by the Test Visualizer subsystem and are automatically posted to the log during the test run, or they can be posted using special operations or scripting methods. The captured images illustrate test commands and simplify the analysis of the automated test results. Exporting Reports An automated testing tool should give users the opportunity to export test results to external files. For instance, it may be useful, when you need to view test results on other computers that do not have this testing tool installed. Also, you should check whether the testing tool can export the test log so that you can send the results to your boss or colleagues. TestComplete stores test results in XML files and allows you to easily export the test log data to several formats (XML+HTML, Excel, MHT), pack the results with WinZip, WinRar or some other archive tool and send the results via e-mail. You can perform these operations from the TestComplete UI or from tests. Also, TestComplete provides access to the stored results. Since the test log uses the XML format, you can parse the results and retrieve the desired information. TestComplete also allows you to create a bug report for test log data and post the report to an issue-tracking system. 8

Additional Needed Automated Testing Tool Features To create powerful software tests, the test recording and playback features may not be enough. Choose a tool that provides many additional features. For instance, you may need the following: A library of useful objects and functions to perform various operations. For example, reading data from and writing it to files, working with databases, working with date and time values, regular expressions and many others. The ability to call Win32 API functions. The ability to call functions that reside in DLLs and.net assemblies. The ability to call routines that reside in Java classes. The ability to work with COM servers. Access to the shell (the WSH scripting object) and the ability to call the operating system commands. OCR functionality to retrieve data from applications windows and controls. The ability to create custom add-ons that will add specific functionality to your software test projects. TestComplete includes all of these features and offers many others. Testing Multi-Language Applications If you have customers throughout the world and your company develops or is planning to develop localized versions of your applications, you have to check whether the software testing tool can work with localized applications and operating systems and whether it supports Unicode. If the tool does not support foreign languages, then, most likely, you will not be able to use it to test localized software. TestComplete supports Unicode and, therefore, it is possible to test multi-language software with it. Automated Testing Tool: Support for Team Work Testers are members of the development team. They work together and communicate with developers, managers and other various people. When choosing an automated testing tool, you may be interested in features that support team work such as concurrent work with project files, sharing script code among several projects and integration with source code control and issue-tracking systems. TestComplete supports all of these features. It allows you to share your test projects with several testers. You can easily link script units from one project to another. You can check project files in and check them out from a source control system directly from TestComplete. Finally, you can add a bug report to an issue-tracking system like Microsoft Visual Studio Team Foundation, Bugzilla, OnTime or JIRA, directly from the test log viewer, and thus notify developers about the problem immediately. 9

Command Line and OLE Automation Support Choose an automated testing tool that you can launch automatically, for instance, at a certain time or after the application build is over. You will be able to do this only if your automated testing tool has the command line or OLE interface where you can specify run parameters. TestComplete supports command line arguments and can work with COM. Other applications, for instance, SmartBear s Automated Build Studio, use these features to call TestComplete automatically. Integration with Team System and Build Software Automated software testing tools are used along with build programs. For instance, the application builder can run tests at the end of each build. When choosing an automated testing tool, check whether it can be used with build programs. TestComplete can be integrated with build software, for example, with SmartBear s Automated Build Studio and with Microsoft Visual Studio, including version 2010 of this popular Microsoft IDE. Also, TestComplete projects can be called from Visual Studio test projects. TestComplete also includes a special assembly that you can use to run TestComplete s software tests from MSBuild projects. Technical Support A very important aspect of selecting an automated software testing tool is the quality of the technical support provided. Prompt responses to your messages prepared by a professional support team will assist in resolving problems and will save you time and energy. Pricing Policy When choosing an automated software testing tool, you have to determine whether its price is reasonable. You should take into account not only the product price, but also the price of add-ons (if they are needed), the support fee as well as training and upgrade prices. To get this information, you may need to call the company s office. Some vendors have special prices that you can only learn about when talking to a representative from their sales department over the phone. SmartBear offers a clear pricing policy. You can find TestComplete prices on our web site: TestComplete Pricing. As you can see, SmartBear offers a feature-rich product at an affordable price. As a result, not only large companies use TestComplete in their everyday work, but also small teams can afford it. Also, you can save money by purchasing TestExecute licenses rather than TestComplete licenses for all of the computers that you are going to run software tests on, but that are not for creating and debugging software tests. Training is a paid option. For more information on this, see our TestComplete Training Web page. 10

Free Trial Version An important factor is the availability of a trial version. Even if the product seems to meet your requirements, the best way to determine this is to download and use it before purchasing. Since you are new to the product, you may get questions and need some assistance with performing tasks. So, another important factor is whether the company provides support for trial-version users. SmartBear provides support for trial-version users. The trial version is a full-featured version, but it has a 30-day limitation and there are limitations on the size of your scripts. Try TestComplete Free for 30 Days Scan and download your free trial to see why testers choose SmartBear for automated testing. You may also enjoy these other resources in the SmartBear Software Quality Series: 11 Best Practices for Peer Code Review Uniting Your Automated and Manual Test Efforts 6 Tips to Get Started with Automated Testing Be Smart and join our growing community of over 100,000 development, QA and IT professionals in 90 countries. 11

About SmartBear Software SmartBear Software provides tools for over one million software professionals to build, test, and monitor some of the best software applications and websites anywhere on the desktop, mobile and in the cloud. Our users can be found worldwide, in small businesses, Fortune 100 companies, and government agencies. Learn more about the SmartBear Quality Anywhere Platform, our award-winning tools, or join our active user community at www.smartbear.com, on Facebook, or follow us on Twitter @smartbear. SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915 +1 978.236.7900 www.smartbear.com 2012 by SmartBear Software Inc. Specifications subject to change. TC-WP-SATT-042012-WEB