My application is too slow, it crashes, and abuses resources now what?

My application is too slow, it crashes, and abuses resources now what? The need for application diagnostics tools Technical white paper Table of contents Executive summary... 2 Application issues viewed from performance testing and post cut-over viewpoints... 3 Performance problem isolation is complex... 4 Traditional methods of triage are not up to the challenge... 5 What if you had an always-on diagnostics tool deployed along side your applications?... 6 HP Diagnostics software... 7 Key features and benefits... 8 HP Business Technology Optimization (BTO) software... 8 HP Performance Center... 9 Key features and benefits... 9 HP Application Performance Management... 10 Key features and benefits... 11 Better business outcomes through application lifecycle management... 11 Discover the benefits of HP Diagnostics for yourself... 12 Contact information... 13

Executive summary In today s business environment, the need to deploy mission-critical business applications quickly is vital. Business agility is a key reason and companies are investing in Service Oriented Architectures (SOA), composite applications, and distributed environments. At the same time, business is demanding more stringent requirements and service level agreements (SLAs) of its applications. Standards, such as web services, help make composite applications easier to build increasing their appeal. These technologies increase the rate of change in complex, distributed heterogeneous environments. Unfortunately, applications do not always produce the desired business outcomes. Nor do they always meet performance and availability expectations despite detailed plans and best practices that have been followed in the application lifecycle. The Standish Group s CHAOS Summary 2009 report indicates that 44 percent of IT projects are late, over budget, and/or with less than the required features and functions 1 It is estimated that the cost to fix an application defect after deployment is more than 150 times the cost to fix in the development phase 2 Application performance issues can appear both before and after deployment into production. Pre-deployment challenges: Triage of application performance issues is complex, and often requires skilled resources with deep application knowledge. Rolling out applications into production without proper testing and triage can eventually cause issues in production. On the other hand, diverting resources to triage issues discovered during load testing can cause delays and impact the applications cut-over date. Line of business plans can be adversely affected by these delays. The snowball effect of diverting resources to find and fix issues can easily impact other ongoing application development projects. Post-deployment challenges: After a business application is placed into production, application issues can result in costly downtime and lost business. Business applications that are exposed to external customers are highly visible extended business outages are usually reported to the media and with negative consequences. Internal-facing application issues can result in reduced employee productivity, which can negatively affect the business s responsiveness to its customers or simply add extra costs that affect the viability of the business. Without proper diagnostics tools, it is difficult for application developers or other IT professionals to isolate the source of these performance and availability issues. Traditional approaches that were once useful for monolithic applications are not viable given today s complex distributed applications that include multiple tiers and SOA-based architectures. Simply throwing more hardware at an application to cover up for poor performance is costly and still presents a significant business risk since the problem isn t truly resolved, just masked. The need for deep diagnostics tools: This problem domain requires deep application diagnostics tools. Diagnostics involves monitoring, tracing, and resolving performance issues, problems, and anomalies across the system pinpointing bottlenecked components. 1 The Standish Group: http://www1.standishgroup.com/newsroom/chaos_2009.php 2 Barry Boehm, Equity Keynote Address March 19, 2007 2

Diagnostics tools give organizations a powerful method to peer into the application tiers as it relates to the individual systems and sub-components. These tools can provide both historical context and application-specific details at the time of an issue. This provides traceability into issues and helps shorten the mean time to resolution (MTTR) shortening the time it takes to repair an application issue and get the application back into service. HP Diagnostics software isolates application performance problems and reduces the MTTR of application performance bottlenecks. It provides actionable information to resolve performance problems. HP Diagnostics includes four main capabilities: Application monitors for gaining visibility into the application and its surrounding environments Discovery of composite application components including web services, application servers, databases, servers, SAP components, and the dependencies among these components including synchronous (HTTP(s), SOAP, RMI, CORBA, REST,.NET Remoting), and asynchronous communication (JMS) protocols Transaction breakdown for taking the end-to-end transaction and peeling back the layers to understand where the time is spent Deep diagnostics for resolving intermittent problems, data-dependent issues, memory leak/thrashing, and deadlock/synchronization-related problems The combination of these four capabilities helps solve the toughest code or configuration issues in preproduction, production, staging, and testing environments, helping to significantly compress testing and tuning cycles, increase productivity, and accelerate performance problem diagnosis and repair. Application issues viewed from performance testing and post cut-over viewpoints Let s examine two points during the application lifecycle where application performance and availability issues will likely surface. The two points in time that we will examine are a) the application performance testing phase and b) a point in time after the application has been deployed into production. Two commonly heard situations are: 1. Pre-production performance testing situation: Our developers and testers are top-notch, our hardware platform is current, and we are using the latest middleware and design methodologies. However, our testers are at a loss to explain why the application test results are so poor including slow transactions, periodic failures, and a crash. 2. Post deployment situation: Our new application was recently cut-over to support the business. The application passed all functional and load-testing benchmarks. But we are getting help-desk reports of hung user sessions, slow transactions, and our IT operations team has been restarting it daily due to countermand memory leakage. The business that is served by this application is being impacted and our IT resources are being diverted to find the source of the application problems usually resulting in unpleasant blame game style conference calls. The issues encountered during performance load testing and after deploying applications to live environments need to be overcome and be resolved quickly. The business depends on applications being deployed on time and supports its critical business services for the end users with a high degree of availability. From either viewpoint, the impact of application issues can be high. 3

Performance problem isolation is complex The business services rendered by today s custom applications are by nature, complex, and cross multiple boundaries (physical, logical) over multiple data centers, IT groups, or departments, and in many cases leverage external third-party applications/services in fulfilling the functionality. Whether it is during the testing phase or after the cut-over of a new application, performance bottlenecks and other application issues can occur. The general categories of these issues can be summarized by: Business transactions are taking too much elapsed time and not meeting performance objectives. When there are multiple application tiers (for example, presentation tier, business logic, and database) including applications that are composed of sub-services, it is hard to focus triage efforts at the right portion of the application. Within a single tier there are many application artifacts and tools are needed that can isolate issues to the code level. Application locks up and stops accepting new service requests. It can be hard to determine where within the distributed application the problem is occurring. In some cases the middleware logs may show a sub-component that is failing but it may be hard to relate to the original business service request. Application exits unexpectedly. The middleware or operating system (OS) dump will indicate where the application exited but does little to provide business context or what user activity was involved. Excessive resource usage is reported by IT monitoring teams. CPU and memory usage/leaks are common complaints in this category. Without insight into the sub-components of the application it will be impossible to diagnose and correct. Business transactions fail (with software exceptions or improper results) Sometimes resource issues are not handled properly within applications causing them to fail in ways that were not caught in functional test areas. Not all components within your organization that are working together to solve the business problem may have been tested to the same degree. It is often quite easy for load testing teams to spot these issues during their activities since these issues are likely to be indicated by poor test results. Likewise, from the production viewpoint, if the application is being properly monitored with an availability and performance viewpoint (and ideally from an end-user perspective), these issues are easily spotted and can be brought to the attention of IT and application support teams. However, if the application is not properly monitored you will have many calls logged to your help desks when the application misbehaves. Unfortunately, once poor test results have been indicated or the application performance and availability events/tickets from a live application have been raised it is only then that the arduous effort begins to determine the cause of the issue. Remember, it is normally only after this isolation phase has concluded that a remedial solution can be recommended. It normally takes much longer to isolate issues than it does to remediate them so during this phase the clock is ticking and costs are accumulating rapidly. This isolation effort is made more difficult due to the fact that today s applications can be composed at runtime and can depend on many IT systems and existing applications. Some of the direct challenges include: 4

Lack of visibility into application tiers and backend processing leads to the inability to determine root causes of application issues. Additional investigation resources are required from many groups to track down problems, as there is no tool to triage and help discover root causes efficiently. Performance problems will manifest later in production where they are more costly to fix. Traditional methods of triage are not up to the challenge To isolate problems efficiently in complex, composite application environments, your developers and QA teams, as well as your IT operations and application support teams, need comprehensive integrated capabilities that support the triage effort. This triage is best done in the context of an issue s effect on the end users and the business. Point solutions deployed in IT silos cannot deliver the end-to-end view necessary for quick resolution. Approaches for solving application performance and availability issues have existed as long as there have been software developers. Each approach comes with its salient faults: Debug logs: Developers can add extra debug output to their applications in order that they can see what led up to an issue or to see where time is being spent. These can slow the application down and can skew performance characteristics and this approach is not something that would likely be used in a live production environment. Coordinating logging across a complex distributed application is tough and the analysis time of the resultant logs can take extensive manual effort. Attaching debuggers: Developers can attach debuggers and set break points on conditions to examine application internals. This is very hard to do in a multi-threaded and multi-container composite application because you first need to isolate the issue to a particular JVM or.net process. It is highly unlikely that developers or IT teams have deployed these tools to production environments and unlikely to allow stopping live application traffic for debugging purposes. Attempting to reproduce a production issue in a test environment: Recreating the application environment in a non-production environment in the hopes of reproducing the issue. This is what is often done in conjunction with the other approaches listed here. Test environments are not the same as production environments especially data sets or deployment resources (load balancers, memory, CPU, etc). It is difficult to re-produce customer traffic in a test environment usually limited to using the same scripts used to test. Adding resources: Some application resource issues can be masked by adding additional hardware (typically CPUs and memory) at the application or by resetting the application daily. This results in higher IT costs. The business outage risk remains if an application reset cannot be scheduled during times of inactivity. In order to perform effective isolation you need to be able to look at a broad class of problems from a single comprehensive solution, whether it s an application receiving a database error, a slow response time, or an application deadlock. Requiring individuals to learn multiple tools, and invest in different technologies leads to a fragmented root cause analysis (RCA) process. 5

What if you had an always-on diagnostics tool deployed along side your applications? This problem domain requires deep-dive application diagnostics tools. Diagnostics tools give organizations a powerful method to peer into the application tiers and code as it relates to the individual systems and sub-components. If deployed and used continuously, they can provide both historical context and application-specific details at the exact time of an issue. This provides traceability into issues and helps ensure a shortened isolation time leading to a shortened MTTR. Diagnostics tools help you monitor, isolate, and resolve issues, problems and anomalies across your complex distributed applications pinpointing bottlenecked components quickly. Most importantly they provides visibility and actionable data that enables developers, QA teams, operations and application support teams to collaborate, diagnose, and improve applications efficiently. Before we delve into our response to this need, it is vital to understand the types of capabilities that should be available in such a solution. What is needed in a solution to help swiftly isolate performance and availability issues is the following: Ability to monitor and collect data such as: CPU usage and service request latency data down to the application sub-component level (i.e. the class and method level) Historical resource utilization capture over time (heap memory, threads, resource pools, queue depths, and so on) Capture of application flow instances (outliers for instance) including multi-container and distributed flows along with their exceptions Complete transaction traceability and application triage: The ability to gather high-level business transactions data as well as low-level service request flows, so that they can easily be correlated to show the problem occurrence in the performance test or synthetic transaction, as well as the corresponding method or SQL query that is slow Ability to follow service requests across servers and application container boundaries in order to support quick isolation Being able to solve memory leaks, hung threads, and other complex composite application problems, using advanced root-cause analysis Enterprise readiness: Ability to be deployed to production monitoring 24x7, global environment and scalable to thousands of applications Configurability to support both 24x7 monitoring in production but with the ability to drill down and enable additional isolation tools without re-starting the application Role-based security (on a per-application basis) to allow for a variety of user and use-cases (IT monitoring, developer, performance specialist, and load testing specialists) Integration with testing as well as end-user monitoring tools: Integration with testing as well as monitoring tools so that diagnostics data is available along with test and monitoring data A low-overhead solution that can be used both during performance testing and left on during production Diagnostics data should be available through the test/monitoring solution every time, so that tests do not need to be repeated for further actionable information 6

HP Diagnostics software HP Diagnostics software is a tool that fits the requirements discussed earlier. HP Diagnostics software helps you improve application availability and performance in preproduction and production environments helping to significantly compress testing and tuning cycles, increase productivity, and accelerate performance problem diagnosis and repair. You can utilize HP Diagnostics software to drill down from the end user into application components and cross-platform service calls to pinpoint and resolve the toughest problems. This includes slow services, methods, SQL, out-of-memory errors, threading problems, and more. HP Diagnostics software extends HP LoadRunner 3 software and HP Performance Center software capabilities to address the unique challenges of testing and diagnosing even the most complicated, composite applications. It allows you to identify issues early that are often hidden in testing but show up in production, such as a slow memory leak. This same low-overhead solution can follow an application into a production environment and provide the same capabilities within the HP Application Performance Management solution. Figure 1: HP Diagnostics view showing slow server requests 3 HP LoadRunner an industry-leading solution for simulating consistent, measurable, and repeatable load tests from a single point of control that allows you to replace real users with thousands of virtual users. 7

Key features and benefits Drills down from slow, end-user transactions to the bottlenecked component, method, or SQL statement, helping to solve memory, exception, and other common problems more quickly. Reduces MTTR in environments such as J2EE,.NET, ERP, SOA, CRM (Oracle, PeopleSoft, or SAP), and others Streamlines diagnosis, automatically detecting all components touched by a business process and tracing them with no user intervention. Integrates dynamic thresholding and alerting with existing infrastructure management solutions Comprehensive capabilities to isolate performance and availability problems in composite environments: J2EE,.NET application servers, SOA, mainframes (MQ Series, CICS), portals, integration servers, and databases Integrates fully with HP Business Availability Center (BAC) software, HP LoadRunner, and HP Performance Center software, or can also work stand alone. Provides complete application visibility across the application lifecycle, enabling higher application quality when applications go live HP Business Technology Optimization (BTO) software HP Diagnostics integrates fully with HP Performance Center, HP LoadRunner, and HP Business Availability Center software. These solutions are each offered separately but are also tightly integrated to provide a seamless solution. These software solutions are part of the HP BTO approach to application lifecycle management. This approach is a business strategy that extends the principles of quality management across the entire application lifecycle. It directly responds to the increasing pressure on IT organizations to improve business process quality in the face of shrinking budgets and organizational complexity. Along with HP Diagnostics, the following HP BTO software can help your enterprise govern and deliver IT-enabled business processes, and manage their quality, while reducing the overall cost of technology. 8

HP Performance Center HP Performance Center software is an integrated set of software products designed to automate key performance optimization processes, such as testing, tuning, bottleneck identification, stress testing, capacity planning, and diagnostics. It provides a single platform that enables a consistent, repeatable process for performance testing critical applications and business processes. HP Diagnostics integrates with HP Performance Center so that you determine slow or problematic transactions, and drill down to correlate procedure calls with transactions aiding problem isolation. Additionally, and most importantly, analysis monitoring and diagnostics reports are combined so you can, at one glance, see the complete transaction trace and drill down for much easier problem resolution. Figure 2: HP Performance Center has a deep integration with HP Diagnostics Key features and benefits Includes HP LoadRunner Software an industry-leading solution for simulating consistent, measurable, and repeatable load tests from a single point of control that allows you to replace real users with thousands of virtual users Test applications across a broad set of technologies including Web 2.0 and SOA applications Gives an accurate picture of end-to-end system performance before going live. Allows verification that new or upgraded applications meet specified performance requirements Pinpoints end-user, system-level, and code-level bottlenecks rapidly and with ease. Identifies and reduces performance bottlenecks during the development lifecycle Decreases the risk and cost of production issues by allowing testing earlier in the application lifecycle. Reduces performance testing cycle time and total cost of ownership 9

HP Application Performance Management HP Application Performance Management solution is based on the integrated HP Business Availability Center (BAC) set of products and provides a unified dashboard for managing IT operations that help meet business objectives. It lets you view key business process and system indicators in real time, from an end-user, business-level, and service-level perspective, and lets you use a center of excellence so that your production applications meet service-level goals and deliver business results. Gartner Inc. has placed Hewlett-Packard Software and Solutions in the Leaders quadrant of the Magic Quadrant for Application Performance Monitoring (APM) report 4. The report evaluated 19 vendors on completeness of vision and ability to execute. A complimentary copy of the report can be viewed at: http://h20195.www2.hp.com/v2/getpdf.aspx/4aa0-2829enw.pdf Figure 3: HP Diagnostics showing an SOA Service topology isolate problem areas fast 4 The Magic Quadrant is copyrighted 2010 by Gartner, Inc. and is reused with permission. The Magic Quadrant is a graphical representation of a marketplace at and for a specific time period. It depicts Gartner s analysis of how certain vendors measure against criteria for that marketplace, as defined by Gartner. Gartner does not endorse any vendor, product or service depicted in the Magic Quadrant, and does not advise technology users to select only those vendors placed in the Leaders quadrant. The Magic Quadrant is intended solely as a research tool, and is not meant to be a specific guide to action. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. Gartner, Inc.: Magic Quadrant for Application Performance Monitoring, by Will Cappelli, February 18, 2010 10

Key features and benefits Provides 24x7 application and infrastructure monitoring and alerting, diagnostics, periodic health checks, capacity planning, prediction, and trend analysis for larger mean time between failure (MTBF) Includes HP end-user management software, which proactively monitors application performance and availability from the end-user perspective as well as real user monitoring so that you can fix issues before customers notice problems Filters real-time user information to your development and QA teams so you can reduce MTTR, improve service-level performance, decrease application downtime, and lower the total cost of monitoring enterprise applications and systems Makes IT service management (ITSM) incident and problem management processes more efficient and business aligned Accelerates problem isolation and resolution by automating standard operational processes Simplifies the management of complex business transactions and service level agreements across heterogeneous environments in a unified dashboard Better business outcomes through application lifecycle management HP Diagnostics software enables your business to extend its QA process and more easily implement best practices for diagnostics throughout the application lifecycle. By taking a lifecycle approach to performance and availability management, you can avoid the process breakdown common in traditional approaches to software quality. Instead of treating QA as a separate silo organization within the business, you can approach diagnostics and software performance management as a discipline that spans the software lifecycle. This provides an opportunity for cross-functional teams to collaborate and deliver high-quality software services. HP Diagnostics supports various stakeholders across the application lifecycle, from development to testing to production. Because we offer an integrated solution and a common set of tools for the entire software lifecycle, we enable QA engineers to more easily collaborate with developers and production software engineers to help maintain better performance in the production environment. HP Diagnostics Profiler is a light-weight, developer-centric version of diagnostics. It gives developers a view into the Java/.Net instance they are currently working on, so that they can debug issues quickly and easily. The Profiler requires no server install, licensing, or configuration, and can be used by developers on their desktop as a low-overhead profiler, free of charge. (Other profiling tools create huge overhead.) The HP Diagnostics technology is common across the Profiler for developers, pre-production diagnostics for performance testing, and production diagnostics for application performance management. Common technology and knowledge sharing fosters collaboration and a greater understanding between teams. Ownership and accountability is spread across teams and the lifecycle. Data from diagnostics is clear and actionable, and hence there is a clear sense of responsibility. Everyone has a more vested interest in issue resolution. Assets such as requirements and SLAs can be managed more effectively helping teams stay on task and the application exceeds business and user expectations. 11

Figure 4: Application Performance and Availability Lifecycle TESTING OPERATIONS Record business process scripts Test and optimize Bidirectional asset reuse Reuse business process scripts Monitor and optimize Business processes and SLAs from testing reused in production Real user information and SLAs leveraged in pre-production Unified diagnostics/sitescope 1 2 3 4 Mitigate risk with testing and monitoring Increase IT efficiency with requirements and script reuse Increase collaboration to accelerate time to resolution Optimize business performance Discover the benefits of HP Diagnostics for yourself HP and its partners provide a variety of consulting and education services, enabling you to draw on our expertise and knowledge of best practices. We can help you decide what diagnostics solutions would be right for your business needs and work out custom ROI requirements prior to deployment. The HP Diagnostics Profiler for Java and.net can be downloaded for a free evaluation at: www.hp.com/go/diagnostics 12

Contact information To find an HP Software sales office or reseller near you, visit www.managementsoftware.hp.com/buy HP provides high-quality software services that address all aspects of your software application lifecycle needs. With HP you have access to standards-based, modular, multi-platform software coupled with global services and support. The wide range of HP service offerings from online self-solve support to proactive mission-critical services enables you to choose the services that best match your business needs. For an overview of HP Software services, visit www.managementsoftware.hp.com/service Share with colleagues Copyright 2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Java is a U.S. trademark of Sun Microsystems, Inc. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. 4AA0-1852ENW, Created April 2010