Whitepaper: The Cost-effective Performance Engineering Solution How to ensure high application performance with little expense
Table of Contents Cost-efficient application performance management 3 Safeguarding application performance is not expensive 4 Available application performance engineering tools 4 Why not integrate performance measurement tools into the application code? 5 Important aspects of an application that should be monitored 6 inspectit helps you solve your performance problems 8 Fact Sheet inspectit 10 Page 2
Cost-efficient application performance management Happy End? After two years of developing an improved version of the internal business system based on the newest technology and integrating the gathered requirements of the employees, the development lead was happy to announce: The application was deployed successfully and starting this second we are live. We have done it, folks, the hard part is over. After the last few months of working long hours to keep the fixed deadline, the development team was relieved that the hard part seemed to be over. The next day the employees would use the new system and benefit from the added functionality and the improved throughput. Every functionality had been tested thoroughly, so what could possibly go wrong? The next morning, after the first server crashed due to memory problems and a number of employees complained about the poor interface response times and the very long time it took for every single business transaction, the development team realised that the hard part in fact had just began. Until lunch, three additional servers broke down and the flood of complaints grew. Nobody could explain the bad performance and stability problems with the new software. The Pressure grew from hour to hour, management complained that the employees could not work as expected. Wild guesses were being issued by the all stakeholders, each pointing their finger at other departments. After two days during which no work could be done on the system, management decided to switch back to the old system. Thorough performance analysis of the developed solution showed that the planned architecture was and would never be able to handle the number of requests issued by the users. The whole architecture would need to be revised and re-implemented. Most of the code would have to be adapted. Another year passed until the software was deployed again - this time thoroughly tested for stability and performance. Apart from several small migration issues, the system was running smoothly and the the employees were able to benefit from the new functionality and better performance. Performance and stability You can call yourself lucky if you never experienced a scenario like the one just mentioned, but it is well possible that you already have.. Application performance and application stability is growing to become the project killer number one of today s software projects. The same as for functional problems applies to performance and stability issues: The earlier the problem is detected, the cheaper it will be to fix (with respect to time, complexity and money). Unsatisfactory performance often makes it necessary to revise and change the entire architecture - as a result, whole components need to be rewritten and optimized. The later performance problems are detected during the development phase, the impact is higher by far than compared to functional defects. What complicates matters is that the solution of performance problems usually affects many more components of the software than common functional defects. Developing and deploying software without adequate performance management processes can be a considerable risk: maybe it will work, maybe and more probably it will not. During the last few years the importance of application performance engineering is growing steadily. Many projects are still not implementing an adequate environment to detect performance problems early during development. They are still not creating an adequate load test environment and they are still not setting up tool support to gather information about their application during a load test or during production. With the effect, that if a performance problem arises, first the entire supporting environment needs to be set up, before it is possible to find and fix the problem. During our consulting assignments we met a number of developers desperately in need of tool support to analyze their components from a performance point of view without too much overhead. Often requests for performance tools or even for performance testing is denied by management due to the expected high costs. Managers and developers can now breathe a sigh of relief: NovaTec s inspectit will solve their problems for little money. Page 3
Safeguarding application performance is not expensive The stated reason why no performance safeguards are in place is often due to high costs. During project planning, performance engineering activities are often forgotten or postponed shortly before deployment on the production system - a few days prior to going live. Often performance tests are not executed at all at this point since the deadline is close Safeguarding application performance in fact does not cost, but saves money in the long run. Application performance engineering is - like functional testing - a purely riskdriven activity. In fact the sole reason why functional tests are implemented and executed is that the costs (with respect to time, money and quality) are higher if these problems first occur during production by the customer. The budget of functional testing is directly derived from the risk and the impact of functional problems. The same holds for performance problems. It is naive to think that complex applications will somehow automatically provide high performance. By comparing the risks with the impact and taking the reduced risk of performance problems into account when integrating performance engineering, a budget for these activities can easily be granted. Performance engineering is thus not costly, but in fact presents a cheaper and safer approach as it helps reducing the risk of performance problems and thus the risk of even higher expenses (ranging to and including a premature termination of the project!). There is no one fits all performance engineering solution and not every project needs a high-standard performance engineering process. Often performance engineering is confused with performance testing. Performance engineering is a process starting with the project initialization, which continues during the complete development lifecycle and supportsthe whole software lifecycle. Performance testing is one step of performance engineering during which the actual performance tests are carried out. It is invaluable that during the definition of the project goals, performance goals are defined as well. The information gained (number of users on the system, distribution of the load, intents to increase the system size after deployment, detailed information of how long each important use case may take) will help greatly during defining the software architecture and during the implementation. Based on these information tests can be set up ensuring that the performance goals can be reached. For these verifications tools are necessary. Available application performance engineering tools During the last few years suitable costefficient alternatives to expensive load generation tools were provided. Projects that do not need the - no doubt unsurpassed - functionality of the market leaders can still use free, open source tools like JMeter or Grinder for power load tests. But generating load on the application is only the first step. Without the possibility to analyse why and where an application became slow as soon as 50 virtual users perform their predefined work, the root cause of the problem cannot be identified and no optimization can be started. Thus load testing without application monitoring does not give any insights into the application and should be considered as a necessary first step for further analysis. At this point, you know that you will not achieve your goal, but you can still only speculate about the why. We usually see guesswork at this point: It might be this database statement or worse finger pointing between departments or teams: Our component is running just fine, but component X is always this slow. Integrating monitoring within the application will allow the developer to check why the application is getting slower, he is able to directly pinpoint the problem area and knows which part of the application needs to be reviewed and possibly optimized. Until now, cost-effective application monitoring solutions were not available on the market. Thus the project team had to decide whether to use one of the established tools of one of the market leaders or implement own monitoring logic within the code using log output. Unfortunately projects often went the third way and did not implement any performance safeguarding process at all. The performance testing tools that are currently available can be divided into the following categories: Inhouse-implementations and Professional performance solutions with excess functionality Page 4
Why not integrate performance measurement tools into the application code? The short answer: Flexibility and maintainability. You will not be as flexible as you will need to be when trying to inspect your application with respect to performance. The more measurement tools you integrate into your software, the less throughput the software will provide. This is due to the fact that collecting the monitoring information also takes time. In fact, including too much monitoring can get you into a situation in which the application performance is so far reduced that the monitoring will not provide accurate performance data. Performance problems can occur anywhere within the entire software stack: In-house components, thirdparty libraries, the used application server, database queries, the operating system etc. Integrating time measurements only within your application code thus will only cover some of these possible areas. Isolating performance problems usually includes an iterative refining of the gathered monitoring information. The best solution is to first take a look at the functionality that is proven to be slower than expected and add measurement points to the components called within this functionality. If this information is not sufficient, refine the monitoring and include additional measurement points. As soon as the performance problem has been isolated and fixed, at least one monitoring point that easily allows checking for this problem again should be integrated. Then this configuration should be stored and all other measurement points be deleted. With manually integrated code to measure your application components, the time to define, refine and remove monitoring is very time-consuming. Aside from that, the integration of non-business-related code always complicates the readability and thus maintainability. Different open source frameworks provide support to integrate monitoring directly with the code and allow basic visualization of the results. The most well-known open source framework is JaMON. We often see the approach to integrate monitoring and/or performance information with the standard logging process (based on tools like the famous log4j). In most cases this approach is chosen to be able to implement some quick and dirty monitoring facility to fix the problems at hand. We have seen multiple projects in which a small performance logging approach like this grew to become the common approach for the entire project. We strongly recommend not to integrate performance monitoring directly within the source code. It is inflexible and cannot provide all the information you need to find performance problems - in addition, usually only time monitoring can be performed. At the other end of the scale are professional solutions to performance engineering - the commercial tools, like for example CA Wily Introscope or dynatrace. Nearly all tools provide the performance analyst with the necessary functionality to inspect and test an application in respect to performance. But all software solutions in this field also imply high costs. These tools provide unsurpassed functionality, are thoroughly tested and provide support for nearly all environments (even exotic ones). Still many projects just cannot afford to buy an expensive tool to monitor the application and pinpoint performance problems. Due to this NovaTec GmbH proudly announces the release of inspectit. inspectit is a Java monitoring tool which focuses on the core functionality necessary to cope with performance problems and provides transparent application monitoring for load test support. Using inspectit at least 80 % of all performance engineering requirements can be easily verified. Page 5
Important aspects of an application that should be monitored We strongly need to differentiate between monitoring a productive application and trying to pinpoint performance problems. For monitoring an already productive application which does not impose severe performance problems, a minimal set of monitoring points should be defined, thus allowing for trend analysis. The focus of this method is to detect deviations from the usual behavior and fix them before the users of the application are affected by the problem. Less is more is the key. The optimal monitoring set contains as little information as possible to still provide all the necessary information to monitor the application and to find the problem at hand. increase the load on the resources (or increase the amount of necessary resources). As soon as the request for a specific resource exceeds its availability, the request will be put on hold - thus decreasing performance of the application.. Due to this close interdependency between resource and application performance, significant resources need to be monitored. As soon as monitoring shows that resources get scarce, adequate actions need to be taken. Inspect the application With application monitoring you are able to open the blackbox of the Java virtual machine and collect information about your application during runtime. As each application is different the amount and the locations to monitor differ as well. The most common approach is to monitor at least the business-critical interfaces of your components. Depending on your need for information more monitoring points can be inserted to gather more information about your application. For pinpointing existing performance problems or in-depth performance analysis, the amount of measurement points can be increased. For this scenario the tool should help to provide as much information as possible about the internal processes of the application and their performance data - still without causing too much overhead. Gathering all available information is also contra-productive as this will cause too much overhead and falsifes the gathered information too much to be of any help (this is the common problem of profiler tools). There is certain essential information that should always be gathered while testing an application for performance: Inspect the resources All applications need resources during runtime. Increasing the load of an application will most likely also Figure 1: inspectit shows important information about the platform the application is running on The most evident resources are system resources like the CPU and memory utilization or database resources like a database connection. Resources are also often pooled (e.g. in an application server), thus these pools need to be monitored as well. With inspectit the monitoring of system resources will be provided directly out of the box. The inspectit agent directly accesses the platform it is running on and gathers information about the resources of the operating system and about the Java virtual machine. With inspectit you can easily place a time-measuring point on any method of any class within your application (or even all third party libraries as long as they are also written in Java). This measuring point provides you with the information about the frequency of and the execution time of the respected method. Using this approach you can easily monitor the execution count of your components and stay informed about all execution times of your methods. Page 6
By placing a measuring point on the entry method of an user transaction you can directly monitor the time your application needs to answer the user request. This information will - like all information within inspectit - be stored in a database for further analysis. Based on this information trends can be anticipated and actions can be taken. Inspect the application flow For in-depth analysis of use cases it is necessary to get a precise understanding what the application does and which steps within the application flow are slow. Unfortunately, the application flow is often rather complex and spans a huge tree structure. The flow path through this tree is often closely related to the input variables. This often leads to situations in which certain use-cases are infrequently slow or at least differ greatly in their performance depending on the input. For such situations the best analysis mechanism is to treat each invocation of the use case as a separate set of information. To do so, the tool must provide the unaggregated application flow for each call of the use case. inspectit implements this concept with its invocation sequence tracer. This functionality allows to define the starting point of a use case and will track each invocation of this method independently. The Illustration (figure 2) shows an invocation sequence within inspectit that was recorded for a web-based shopping application. This invocation sequence sensor was configured to record each call to the method ordercartforcustomer separately. This method is called by the application as soon as the shopping cart is committed by the customer and the order is to be be shipped. inspectit shows all recent invocations of this method (upper window) and provides the duration of each invocation. The bottom part shows the exact invocation sequence for the selected invocation sequence. The performance analyst can walk through the invocation tree and inspect each step. Using this functionality, the analyst can quickly see that the problem is located in the VIP Customer inspection in ordertocalculatesomebonus discount. Closer inspection shows that in order to check for the VIP status of a customer, many order objects need to be retrieved from the database. In fact this implementation always retrieves all orders of the customer to calculate if the customer has VIP status by summing up all orders of the customer and checking if the overall amount exceeds a given threshold. For relatively new customers the total number of orders is small, thus the application runs smoothly for them. For regular customers - with a large number of orders - the application will become slower and slower. Problems like this are very hard to find without a suitable monitoring tool. In addition to inspecting the application flow, displaying a summary often allows pinpointing problems more easily. Based on the example used in this chapter, the next illustration shows a summary of all methods invoked in this invocation (inspectit also provides a summary for all database queries invoked in the invocation sequence). Using this view one can easily see that the method findbyartikelid is called very often and utilizes nearly 29 seconds. Figure 2: Method summary of an invocation sequence. Page 7
Inspect the database queries Current estimates put unnecessary or inefficient database queries at one of the top-most positions of performance problems. This usually occurs as the database is abstracted by additional frameworks. The developers will never see the actual query executed by the In addition inspectit allows to display SQL information directly within invocation sequences easily showing if a method was slow due to complex and inefficient algorithm or due to invoked database queries. inspectit helps you solve your performance problems NovaTec incorporates years of experience in dealing with application performance ranging from the definition and integration of performance processes to safeguard the development of application to firefights in critical projects to improve performance in a short time. We are proud to offer tool independent performance consulting. To offer the best possible performance consulting we have developed a strong relationship with the two big players in today s application performance markets: dynatrace and CA Wily Introscope. Our engineers provide years of experience in either of these tools and most of the other tools available on the market. Figure 3: inspectit Database sensor: Quickly inspect all your SQL queries for performance framework. Another frequent scenario is based on the fact that defining new database queries for each persistence framework takes up time. Developers thus tend to re-use existing queries even if these queries return far too much information. But writing new queries results in implementing new code, ranging from new interfaces to new data transfer objects. inspectit allows to track all database queries that were created and executed within your application. By default the database tracing is activated out of the box with no manual configuration being necessary. The provided information allows to quickly inspect all queries with respect to their execution frequency and their respective execution time. Since all data is stored within a database, trends become visible. Inspect the architecture Another significant use of monitoring tools is to evaluate the adherence to architecture specification. Instrumenting the entry methods of the layers and components of the application with an invocation sensor allows to easily check which layers and components were used by each use case. This information is invaluable for architects to check whether the dependencies between applications are suitable and to understand which components are currently used to implement a use case. For four years NovaTec engineers of the business unit Application Performance Engineering have been implementing a solution to application performance monitoring and thus we are proud to officially announce the first official release of this application, inspectit. The development of inspectit includes performance monitoring experience gathered from many highly skilled NovaTec engineers to help you solve your performance problems. With inspectit you can easily integrate application monitoring into your development process for less than a tenth of the cost of most other commercial monitoring tool. However inspectit is still absolutely able to ensure that the application will meet the defined performance requirements. inspectit works by transparently integrating callback hooks into the application using load-time Java byte code modification. These hooks are then used by inspectit s sensors to collect various monitoring information about the application. This monitoring approach is integrated with a minimum of (transparent) code adaptation and Page 8
thus is the safest possible way to integrate monitoring. All measurement logic is extracted to the sensors that run outside the business relevant logic code and in this way is safely separated from your application. The flexible and solid core framework of inspectit allows easy integration of additional sensors. Hence new sensors can be created to collect the information you need for your application. An inspectit installation consists of at least one agent running in the application JVM and monitoring the application(s), one (or quite rarely multiple) central measurement repositories (CMR) that store the data sent by the agents to a persistent storage and multiple clients that are connected to the CMR and display gathered and aggregated data in a significant way. The inspectit agent is the component that enables measuring points within the monitored application. It must be integrated into the startup script of your application (or application server) and automatically instruments the configured methods with the defined sensors. All configuration settings are defined within a configuration file residing in the agent configuration folder. The integration of the agent is as simple as adding one additional JVM option to the startup script of your application. Interested? Contact us at info@inspectit.eu or call us at +49 (0)700 5280 5280. You may also use the contact information provided below. M.Sc. Stefan Siegl Business Unit Manager Application Performance Engineering Project lead inspectit stefan.siegl@novatec-gmbh.de M.Sc. Patrice Bouillet Lead architect of inspectit patrice.bouillet@novatec-gmbh.de Setting up the CMR and the GUI is even simpler - just copy the releases to the preferred location and execute the startup scripts. Page 9
Fact Sheet inspectit Application performance and stability is growing to become the project killer number one of today s software projects. Detecting a performance problem in a production environment can result in - in extreme cases - to a complete redesign of the software architecture. Succeed in the performance struggle by opening the JVM black box, enabling you to see which parts of your application perform as expected and which need to be improved. Win the performance battle by integrating transparent inspectit sensors within your code. Integrating measurement points is as easy as writing a single line of configuration defining the location of this measurement point. Save time, effort and customer support by identifying possible performance bottlenecks prior to shipping the application to your customer. The earlier you can identify and fix a performance problem, the better. Features Fast and simple installation and configuration Start implementing your applications without complex and time-consuming tool setup. inspectit can be set up in a matter of an hour. Afterwards you can start defining your measurement points and start the inspection. Seamless integration inspectit integrates itself seamlessly in any Java application by means of byte code instrumentation. During classloading the measuring code is added, so your business code stays exactly the same, no business logic is altered in any way. State-of-the-art technologies and use of official standards inspectit is written completely in Java and only uses officially released and standardized interfaces. All components of inspectit are based on well-known and standardized libraries like Spring, Eclipse RCP, PicoAgent,... Decentralized architecture The flexible decentralized architecture allows having one (or multiple) central servers collecting, storing and aggregating the data of multiple agents and providing this data to multiple standalone clients for analysis. Optimized for low overhead inspectit is designed to run with as little overhead as possible even in complex clustered environments. Monitoring Functionality inspectit can be used to monitor your live application and provide invaluable information about the current status of your application. Page 10
Flexible architecture allows fast integration of sensors inspectit is based on an highly flexible agent architecture that allows seamless integration of sensors into the application code. A vast number of supported sensors for instant usage is available out-of-the-box. Additional sensors can easily be integrated. Currently inspectit provides the following sensors: Time measurement (Timer sensor) Easily monitor the execution times of methods within your application to see which methods use up the most time. The sensor is also able to provide the pure CPU times which allows checking whether the method took long due to complex logic or if it was merely on hold. Platform Sensor The platform sensor allows monitoring of system resources like CPU utilization, memory utilization, threading etc. Database Sensor Performance problems are often related to inefficient or unnecessary database queries. inspectit allows to monitor the duration and structure of all database queries and provides the query as it is sent to the database even with all parameters flled in. Invocation Sequence Sensor Use invocation sequences to get unaggregated call invocation trees. Using this sensor you will directly get the information which methods were called by the users and which methods took long within this execution of the logic. This functionality is invaluable for inspecting invocations that are slow sporadically or in-depth analysis of certain functionality. Flexible possibilities for sensor configuration inspectit incorporates a flexible configuration of sensors. Architecture Validation Architects can focus the instrumentation to gather architecture validation information, allowing them to verify if the developers use the architecture as the architect designed it to be used. Visualization A highly configurable state-of-the-art user interface allows easy analysis of the gathered data. Support for all Java applications on all platforms inspectit can monitor all Java applications (JVM version > 1.3) and works with all known application servers. The most cost-effective Java monitoring solution inspectit provides all functionality necessary to implement application monitoring for Java applications but costs only a fraction of other tools. Page 11
2009 NovaTec - Ingenieure für neue Informationstechnologien GmbH All other company or product names are the trademarks or registered trademarks of their respective holders. Version 1.1 / Sep 2009