UNIT TESTING FOR PERFORMANCE USING NTIME. Author: Ramajayam Ramalingam

Transcription

1 UNIT TESTING FOR PERFORMANCE USING NTIME Author: Ramajayam Ramalingam

2 TABLE OF CONTENTS AN INTRODUCTION TO NTIME 2 PERFORMANCE ENGINEERING AND NTIME 3 TESTING FOR APPLICATION PERFORMANCE USING NTIME 4 NTime Test Class Setup 4 NTime Possible Performance Test Scenarios 4 Sample Implementation 5 Setup a.net Console Application 6 Create Entities/Classes to be Tested 6 Write Performance Test Scripts Using NTime 7 Testing & Seeing Performance Results in NTime 8 CONTINUOUS INTEGRATION RUNNING NTIME IN MS BUILD 9 Creating NTime Task for MSBuild 9 Integrating NTime Task with MSBuild 9 BENEFITS OF NTIME 11 CONCLUSION 11 REFERENCES 11

3 The performance of a software application is critical in ensuring user satisfaction and meeting service-level agreements (SLAs). Therefore, performance testing plays an important role in the development life cycle. Performance needs to be engineered throughout the software development lifecycle. Performance goals must be defined and metrics must be gathered and analyzed at each phase of the development life cycle to ensure that performance goals are being met. This whitepaper introduces NTime, a performance testing tool that helps test performance goals throughout the development process and measure key performance metrics, such as the duration of a method call, number of hits to a method and code optimization levels. This paper also demonstrates the need for NTime and explains how to execute a performance test for.net applications using this freely available open source tool. AN INTRODUCTION TO NTIME NTime is a free, open source, performance benchmarking tool created by Adam Slosarski, to write performance tests for.net applications. NTime ensures repeated performance testing of an application at desired intervals. It can test against the duration or number of hits to a method or make use of Windows Performance Monitor probes to check for optimized code solutions. NTime is not a profiler, but a quick way to set time or performance boundaries on.net applications which are susceptible to performance bottlenecks. In other words, an upfront performance specification/metric is required to implement NTime. NTime s source code, installer and tutorials are freely available at ( They can be downloaded and used for measuring and achieving performance goals of any application. NTime can be used by both system architects and.net developers. Architects Architects identify key scenarios for performance, translate them to performance goals for each layer and hand over the minimum performance measures of modules to the developers tasked with NTime testing..net Developers.NET Developers would be interested in the diverse features offered by NTime for unit testing the performance of their modules, which in turn will help them realize the minimum required performance. UNIT TESTING FOR PERFORMANCE USING NTIME 2

4 PERFORMANCE ENGINEERING AND NTIME Performance, as a quality of service requirement, has to be engineered and cannot be planned/executed overnight. During different phases of the development life cycle, the following performance principles must be applied: Gathering Requirements In this phase, performance objectives are set by the development team after consultations with the customer. Units of measure, such as CPU time and memory footprint, are identified. Designing for Performance Key performance scenarios, along with the workload, are identified in this phase. Examples of scenarios include meeting a performance goal or addressing a potential threat to the overall performance. Implementation In this phase, performance requirements are implemented in line with established design guidelines. The performance-verifying unit test cases are created when bottlenecks are expected or a performance scenario s functionality keeps changing. Verification/Testing Regular testing of the code for performance ensures that the final performance goals are met. Various tools should be used to test the scenarios identified for performance. Metrics collected for the units of measure from these tools are compared with the performance goals. Performance Engineering and NTime The performance goals that are agreed upon in the requirements gathering phase are translated to measurable units, such as resource utilization and response time. These units are the inputs to the NTime testing scripts. They should be further translated and used as duration, number of hits and performance counters. During the implementation phase, test cases are written in NTime in accordance with the performance specifications for the identified unit test scenarios. Just like NUnit, NTime can be repeatedly run to verify that, in spite of the changes the developers have made to a particular function, it still runs within the performance limits. NTime, if kept open alongside the IDE, reloads the updated tests automatically, as and when the application is built. Figure 1: NTime GUI 3

5 TESTING FOR APPLICATION PERFORMANCE USING NTIME NTime Test Class Setup To set up the performance test environment before and after the test execution, NTime can be configured using the following attributes: TimerFixture: Indicates that a class is to undergo performance testing. TimerFixtureSetup: Specifies a global set-up environment for the entire performance test. The content of this block is executed before the first test is started. This is like a constructor for a class. TimerFixtureTearDown: Specifies a global teardown environment for the entire performance test. This block will be executed after all tests are executed/completed. This is like a destructor for a class. TimerSetUp: Specifies a place to locally set up the environment for a single test. Before each and every performance testing method starts, this block of code is executed and necessary parts/objects are initialized. TimerTearDown: Specifies a place to locally tear down the environment for each performance test. After a test method has finished its execution, contents in this block are executed at the end of each test. TimerIgnore: Instructs the performance test tool to skip performance test and mark it as warning information. Syntax: Gives an overview of how these attributes should be used with proper syntax. NTime Possible Performance Test Scenarios Using NTime, the following performance scenarios for an application can be verified: I. Duration Testing Duration testing involves testing the time taken for completing a function call. This can be achieved using the TimerDurationTest attribute. For example: UNIT TESTING FOR PERFORMANCE USING NTIME [TimerDurationTest(50, Threads = 10, Unit = TimePeriod.Milliseconds)]public static void searchperformancetest() DemoCustomer customerservice = new DemoCustomer(); IEnumerable<Product> searchresult = customerservice.searchproducts( Loan ); foreach (Product product in searchresult) string result = ID : + product.id; result += ;\nname : + product.name+, + product.description; Console.WriteLine(result); 50 denotes the expected duration of this call. The unit parameter specifies the unit of duration, which could be seconds, milliseconds or minutes. 10 indicates the number of threads being used for this test. When the test is run in NTime, if the actual duration is 50 milliseconds or less, the test is shown in the NTime UI as Passed, otherwise it is shown as Failed/ Rejected. II. Hit Count Testing Hit count testing measures the number of times a method can be run/hit within a specified timeframe. This can be achieved in NTime using the TimerHitCountTest attribute. For example: [TimerHitCountTest(25, Threads = 3, Unit = TimePeriod.Second)] public static void loginperformancetest() DemoCustomer customerservice = new DemoCustomer(); string result = customerservice.makelogin(); Console.WriteLine(result); 4

6 25 is the expected number of times this method will be invoked in the given time period. 3 indicates the number of threads that NTime should use to call this method. NTime creates 3 threads and makes calls to this method. If the method has been invoked for 25 times or more within a second, this method is marked as Passed, otherwise the method is marked as Failed. III. Counter Testing Counter testing is where the Windows Performance Monitor (WPM) probes can be utilized in NTime tests to measure the system resources consumed by the application. TimerCounterTest attribute is used to achieve the same. [TimerCounterTest( CategoryName, CounterName, InstanceName = Instance, Threads = int, MinimumValue = int, MaximumValue = int)] CategoryName is any of the Performance Objects that one can see in the PerfMon counter (such as Processor or Memory). CounterName is the specific counters within the CategoryName, such as Pages/sec or % Processor Time. To gather performance-monitoring statistics, methods need to exercise the TimerCounterTest attribute. loginperformancetest(); searchperformancetest(); Sample Implementation Business Scenario ABC Bank is developing an application to handle a large amount of customer data. ABC Bank s top priority in the application is to achieve maximum performance gains. The application should be able to handle large hits from several users, run fast and utilize the minimum number of resources possible. The application s performance specifications include the following: 1. Application should support 50 concurrent users signing in per second 2. Search module should return results within 2 seconds at a peak load of 10 concurrent searches. 3. For any given operation or operations combined, CPU utilization should not exceed 70% for 1 user Performance Testing Scenarios The following sample performance test implementation intends to measure the performance requirements using NTime: 1. To determine whether the application meets the concurrent access load, the number of hits will be measured 2. To determine whether search results are returned within the agreed upon time, the duration of method calls will be measured 3. To determine the CPU utilization, windows performance counters will be measured Note that the normal execution of a method call might have to be prolonged to ensure that enough data is gathered. The following example tests are used to ensure CPU utilization rates are within 1 to 70 percent. [TimerCounterTest( Process, % Processor Time, Threads = 1, InstanceName = NTimeGUI, MinimumValue = 1, MaximumValue = 70)] public static void loadperformancetest() 5

7 Implementing NTime: Walkthrough Database Setup Create a database named BankDB with the following two tables and host it in your Database Server. g. In the Entity Data Model Wizard, select the Yes option and press Next button. h. From the tables listed by expanding the Tables node, select the Customer and Product tables and then press the Finish button. Now your Figure 2: Database Bank DB Setup a.net Console Application 1. Create a.net console application 2. Add the following dll references to your projects: a. NTime.Framework.dll (at folder \NTime\ Bin\ for.net Framework 3.5 and below. For.NET Framework 4.0 and above, use Recompiled version of NTime Source Code for corresponding version). 3. To use the Entity Framework as the data source: a. Select Add new Items by right clicking the project. b. Pick the ADO.NET Entity Data Model, name it Bank DataModel and press the Enter button. Consequently, select Generate from Database for creating the models. c. Follow the following sequence to set the data provider: New Connection -> Press Change Button -> Select Others as Data source -> Select.NET Framework Data Provider for SQL Server as the data Provider. d. Type your database server machine name where you have created and saved the database BankDB in the Server Name text box. e. Use the SQL Server Authentication with the appropriate User Name and Password that can be used to connect to the database server. f. Select the BankDB as the Database and press Ok button. Entity Data Model is ready to be used by the Application. Create Entities/Classes to be Tested 1. Create a Customer Data Service Interface to interact with the database (through the above Entity Framework). This is called the Data Access Layer. public interface ICustomerDataService IEnumerable<Product> SearchProducts(string productname); Customer Login(string username, string password); public class CustomerDataService:ICustomerD ataservice public IEnumerable<Product> SearchProducts(string productname) return (from product in (new BankDBEntities()).Products where product.name.tolower(). Contains(ProductName.ToLower()) select product); public Customer Login(string username, UNIT TESTING FOR PERFORMANCE USING NTIME 6

8 string password) try return (new BankDBEntities()).Customers. Where(cust => cust.username == UserName && cust. Password == Password).Single(); catch (Exception exception) return null; 2. Create a class (named DemoCustomer) that performs two operations, namely makelogin and searchproducts as the following: class DemoCustomer public string makelogin() string result; CustomerDataService service = new CustomerDataService(); Customer loggedcustomer = service. Login( Ramajayam, rama ); if (loggedcustomer == null) result = Invalid Login Information Entered\n ; else result = Log-in Successfull\n ; result += User Name: + loggedcustomer.username + ;\nfullname: + loggedcustomer.fullname; result += ;\naddress: + loggedcustomer.location+ \n\n ; return result; public IEnumerable<Product> searchproducts(string searchterm) searchterm = searchterm.tolower(); CustomerDataService service = new CustomerDataService(); return service.searchproducts(searchterm); Write Performance Test Scripts Using NTime 1. Add the NTime.Framework.dll Assembly file as Reference to your project. 2. Write a class which is going to performance test the above operations as indicated below: [TimerFixture] public class BankPerformanceTest //Test if the 50 logins could be achieved within a second, assuming there could be an average of 10 threads. [TimerHitCountTest(50, Threads = 10, Unit = TimePeriod.Second)] public static void loginperformancetest() DemoCustomer customerservice = new DemoCustomer(); string result = customerservice. makelogin(); Console.WriteLine(result); //Test if the search method is executed within 2 seconds for 10 concurrent searches [TimerDurationTest(2, Threads = 10, Unit = TimePeriod.Second)] public static void searchperformancetest() DemoCustomer customerservice = new DemoCustomer(); IEnumerable<Product> searchresult=customerservice. searchproducts( Loan ); foreach (Product product in searchresult) string result = ID : + product.id; result += ;\nname : + product.name+, + product.description; Console.WriteLine(result); 7

9 // For a combined operation of login and search, the CPU utilization should not be more than 70% [TimerCounterTest( Process, % Processor Time, Threads = 1, InstanceName = NTimeGUI, MinimumValue = 1, MaximumValue = 70)] public static void loadperformancetest() loginperformancetest(); searchperformancetest(); 3. Now compile and run the project. 4. The application is now ready for performance testing using the NTime tool. Testing & Seeing Performance Results in NTime The procedure for performance testing on the application using NTime is as follows: 1. Open the NTime.exe file and click the Add Reference option. Then browse and select the output Assembly file of the above application. 2. After adding the application s Assembly file into the NTime tool, press the START button/option shown in the NTime UI. 3. The performance test results will be displayed, indicating whether the methods were accepted or rejected based on the performance specification of the test methods. 4. If a performance test method is rejected, then refactor the method until it achieves the specified performance level. 5. Only after all the performance test methods are accepted will the application have met the agreed upon performance specification. The following snapshot shows the results in the NTime UI: Figure 3: Snapshot of NTime Test Result UNIT TESTING FOR PERFORMANCE USING NTIME 8

10 CONTINUOUS INTEGRATION RUNNING NTIME IN MS BUILD NTime does not have any built-in support for running through automated builds, requiring you to write your own task to be used in the MSBuild tool. Once you create the task, it can be integrated into MSBuild. Creating NTime Task for MSBuild The following steps show how to create a Task for NTime that will be used in the MSBuild: 1. Create a Class Library Project in Visual Studio 2. Add the following Assemblies to this project NTime.Framework.dll 2.2. Microsoft.Build.Utilies.dll (Version: 4.0) 2.3. Microsoft.Build.Framework.dll 3. Create a class that inherits the Parent Class Task and override the Execute() method: public class NTime:Task public override bool Execute() if (AssemblyFullPathValue == null) Console.WriteLine( \nassembly Name must be entered\n ); return false; Program p = new Program(); p.domain = new AssemblyDomain(); p.open(assemblyfullpathvalue); if (p.invalidassembly == true) Console.WriteLine( ); return false; if (p.rl.hastimerfixture == false) Console.WriteLine( No TimerFixture attributed Class found ); return false; p.run(); if(stoponrejectedtest==true) if (p.rl.rejectedtests!= 0) return false; Console.WriteLine( ); return true; [Required] public string AssemblyFullPathValue get; set; public bool StopOnRejectedTest get; set; Integrating NTime Task with MSBuild Follow these steps to integrate the NTime task with MSBuild: 1. Compile the above Ntime task into NTimetask.dll. Copy NTime.Framework.dll and NTimeTask.dll into a specific folder, viz. <<NTIME_LIB>>. 2. Write a Build file that will be input to MSBuild.exe as the following: <Project DefaultTargets= NTimeTarget xmlns= developer/msbuild/2003 > <UsingTask AssemblyFile= <<NTIME_LIB>>\ NTimeTask.dll TaskName= NTimeTask. NTime /> <Target Name= NTimeTarget > <Message Text= Starting the Performance Testing... /> <NTime AssemblyFullPathValue= You rassemblypath1\assemblyname1.dll ContinueOnError= true /> <Message Text= Performance Testing Ended... /> <OnError ExecuteTargets= ErrorTarget /> </Target> <Target Name= ErrorTarget > <Message Text= One Of the Build Tasks had Failed. /> </Target> </Project> 9

11 Assume that you have saved this build file as: NTimeBuild.build at the location <<BUILD_ FILE_FOLDER>> MSBuild.exe <<BUILD_FILE_FOLDER>>\ NTimeBuild.build /t:ntimetarget 5. Execute the above command by pressing the Enter key. All the assemblies mentioned in the build file will be performance tested and results will be displayed as shown in Figure Open the Visual Studio Command Prompt tool. 4. Type the following command in that tool: Figure 4: Snapshot of NTime Result in MS Build UNIT TESTING FOR PERFORMANCE USING NTIME 10

12 BENEFITS OF NTIME NTime allows the development team to get a preview of what the performance of a particular scenario would look like. Early diagnosis of a performance bottleneck will reduce effort in later stages when only profiling could identify the bottlenecks. It also enables an architect to performance budget a scenario and give the developer meaningful goals to achieve, since he/she now has an option to verify that the scenario is functioning well within the performance budget. Integrating NTime into a continuous delivery framework will ensure that the frequent changes to the code will not create surprises later in development or during the release of the product. NTime results could also be part of the exit criteria for development. Doing so will provide the reviewer with vital statistics of the code performance (or lack thereof). CONCLUSION NTime, as noted before, is neither a performance benchmarking profiler nor a full-fledged performance testing suite. It does not replace end-to-end performance testing tools like Load Runner. Instead, NTime, as a tool, can be used to generate performance metrics at individual and unit-level source code. It helps the developers to determine whether the source code is ready for performance test. References 1. Section Creating Performance Bottleneck tests with NTime- Windows Developer Power Tools. 2. Code Project article, NTime- Performance unit testing tool, written by Adam Slosarski. 11

13 About Sapient Global Markets Sapient Global Markets, a division of Sapient (NASDAQ: SAPE), is a leading provider of services to today s evolving financial and commodity markets. We provide a full range of capabilities to help our clients grow and enhance their businesses, create robust and transparent infrastructure, manage operating costs, and foster innovation throughout their organizations. We offer services across Advisory, Analytics, Technology, and Process, as well as unique methodologies in program management, technology development, and process outsourcing. Sapient Global Markets operates in key financial and commodity centers worldwide, including Boston, Chicago, Houston, New York, Calgary, Toronto, London, Amsterdam, Düsseldorf, Geneva, Munich, Zurich, and Singapore, as well as in large technology development and operations outsourcing centers in Bangalore, Delhi, and Noida, India. GLOB For more information, visit sapientglobalmarkets.com Sapient Corporation. Trademark Information: Sapient and the Sapient logo are trademarks or registered trademarks of Sapient Corporation or its subsidiaries in the U.S. and other countries. All other trade names are trademarks or registered trademarks of their respective holders. UNIT TESTING FOR PERFORMANCE USING NTIME

14 Global Offices Amsterdam Sapient Netherlands B.V. Damrak LM Amsterdam Tel: +31 (0) Düsseldorf Hammer Straße Düsseldorf Germany Tel: +49 (0) Munich Kellerstraße München Germany Tel: +49 (0) Bangalore Salarpuria GR Tech Park 6th Floor, VAYU Block #137, Bengaluru Karnataka India Tel: +91 (080) Boston 131 Dartmouth Street 3rd Floor Boston, MA Tel: +1 (617) Calgary 888 3rd Street SW Suite 1000 Calgary, Alberta T2P 5C5 Canada Tel: +1 (403) Chicago 30 West Monroe, 12th floor Chicago, IL Tel: +1 (312) Delhi Towers D & E, AL MARKETS DLF 1 Cyber Greens DLF Phase III Sector 25-A Gurgaon Haryana India Tel: +91 (124) Geneva Sapient Switzerland S.a.r.l. Succursale Genève c/o Florence Thiébaud, avocate rue du Cendrier Geneva Switzerland Tel: +41 (0) Houston Heritage Plaza 1111 Bagby Street Suite 1950 Houston, TX Tel: +1 (713) London Eden House 8 Spital Square London, E1 6DU United Kingdom Tel: + 44 (0) Los Angeles 1601 Cloverfield Blvd. Suite 400 South Santa Monica, CA Tel: +1 (310) New York 40 Fulton Street 22nd Floor New York, NY Tel: +1 (212) Singapore Air View Building # Peck Seah Street Singapore Republic of Singapore Tel: +65 (3) Toronto 129 Spadina Avenue Suite 500 Toronto, Ontario M5V 2L3 Canada Tel: +1 (416) Zürich Sapient Switzerland GmbH Seefeldstrasse Zurich Switzerland Tel: +41 (58)