Optimizing Application Development A Key to Business Success

Optimizing Application Development A Key to Business Success best practices WHITE PAPER

Table of Contents Executive Summary...1 Application Problem Resolution, the Major Consumer of Development Effort...2 > Application Development...2 > Application Maintenance...3 The Application Problem Resolution Process...3 > Gather Problem Information...4 > Re-create Problem...4 > Analyze Root Cause...4 Leveraging Technology to Streamline Application Problem Resolution...4 > Streamlining the Development Stage...4 > Streamlining the Maintenance Stage...5 Solution Criteria...5 > Support Entire Application Lifecycle...5 > Support Applications Across the Enterprise...5 > Provide Thorough, Yet Unobtrusive, Data Capture...6 > Integrate Easily with Other Systems...6 Compelling Business Benefits...6 > Increase Business Agility...6 > Reduce Development Costs...6 > Reduce Defects...6 > Reduce Hiring Requirements...6 Some Real-World Examples...6 > Financial Services...6 > Healthcare Software Provider...7 > Retail...7 Conclusion...7

Executive Summary As IT becomes increasingly enmeshed in business processes, organizations are demanding more business applications in shorter time frames. To meet the demand, many organizations are striving to improve their application development processes. This has been traditionally achieved through adoption of agile development methods, implementation of various lifecycle management tools, and off-shoring part of development, as well as through other approaches. Traditional means for development optimization, however, do not address one key process that is a major consumer of time and effort on the part of developers application problem resolution. It has been estimated by analysts and other industry experts that software developers spend 30 to 60 percent of their time identifying and correcting defects reported by the quality assurance (QA) group during the testing phase and escalated by support staff. This process, referred to as application problem resolution, is common to all development organizations, regardless of their industry, development method, and platform. An innovative technology has emerged that addresses application problem resolution head-on. Solutions based on this technology can help save up to one half of the time developers spend resolving problems by automating problem resolution processes over the entire application lifecycle. As a result, the development staff can deliver more applications in shorter time frames, without jeopardizing functionality or quality. What s more, by substantially improving developer productivity, an application problem resolution solution can reduce the number of developers needed to meet the ever-shrinking application delivery times required by the business. That means fewer developers that need to be hired to meet schedule commitments or replace developers that leave the organization through attrition. The combination of advantages achieved through implementing an automated application problem resolution solution gives the business a decided edge that helps ensure greater success. This paper: > Describes the problem resolution process and the demands it places on application developers > > > Discusses how new technology can reduce the time developers spend on problem resolution Presents the criteria for a solution that addresses these issues Describes the resulting business benefits and examples of organizations that have achieved them

Application Problem Resolution, the Major Consumer of Development Effort The demand for business applications is increasing, while application development teams are also under pressure from senior management to reduce application development costs. Complicating the problem is the increased proliferation of multi-tier, Web-based applications and the rapid emergence of service-oriented architecture (SOA). While constructing such highly distributed applications is easier, isolating and pinpointing problems is increasingly more difficult due to the great number of moving parts and constant changes. As a result, application development managers are always faced with making tradeoffs in application functionality, quality, and time to market. In addition, development managers face the problem of having to hire and train skilled programmers, either to speed up a project that is getting behind schedule, or to backfill developer positions vacated through attrition. to agile development methodologies, or are considering it. These methodologies minimize risk by developing software in short time boxes, called iterations, that typically last one to four weeks. Yet, none of these initiatives addresses the time-consuming, inefficient process that is common to all development organizations application problem resolution. This process involves significant developer time in both the application development and application maintenance stages of the development lifecycle. Application Development The various activities carried out in the development stage of the application lifecycle may be grouped as following: > Requirements gathering > Design > Coding > Testing To address these issues, IT organizations are taking a close look at their application development processes and searching for ways to improve them. This quest is of particular importance to senior application development managers, whose success is measured by their ability to deliver needed applications that meet functional and quality requirements on time and under a growing pressure to cut costs. Consequently, some application development managers are now off-shoring a part of their development projects. Others are investing in application lifecycle management tools that allow for better management of software development projects. Many organizations have also either turned Typically, more than half of the total software project time is spent in testing. This involves a wide variety of tests, such as unit testing (usually done by the developer), as well as testing for integration, component, system, system integration, user acceptance, and implementation. The purpose of testing is to uncover and resolve as many problems as possible before the application is released to production. That means that just about all of a developer s time spent in testing is spent diagnosing and investigating defects reported by the testing team. Because testing accounts for more than half of the time a developer spends in the application development stage, streamlining problem resolution in testing will contribute heavily to speeding the application development process. % Time Spent on Problem Resolution 100% 50% 0% Design and Coding Unit Testing System Testing User Acceptance Testing Lifecycle Stage Figure 1. The hidden problem resolution process Problem Resolution Time Maintenance

Application Maintenance Developers may also get involved in the resolution of problems detected in the maintenance stage of the application lifecycle. Problems discovered here do not have as much impact on developers time as those discovered in the development stage. That s because far fewer problems are detected in the maintenance stage than in the development stage. Moreover, not all of the problems encountered in maintenance require developer involvement, as many problems are resolved by the application support team. Problems in the production environment, however, typically have greater impact on the business than those discovered in the development stage, and they can distract developers from their primary task of developing new applications. Not only that, but these problems are more difficult to diagnose than problems detected in the QA lab. A study by Ziff- Davis/The Strategy Group for BMC Software revealed that when reacting to IT incidents, most companies are employing manual processes that require significant human intervention. On average, 14 staff members are involved in responding. to and repairing service outages 1. As a result, application development optimization must also encompass problem resolution in the maintenance stage. The Application Problem Resolution Process A closer look at application problem resolution reveals areas that present opportunities for improvement. The problem resolution process consists of two steps: root cause analysis and problem fix. Typically, the vast majority of the time that developers spend in application problem resolution is spent on root cause analysis. Root cause analysis presents a challenge because the symptom does not usually point to the source of the problem. For example, during integration testing, a particular transaction may not execute. Potential causes of the failure are many and varied, and may include a source code bug, an application server service failure, a network problem, improper configuration, or a combination of these. As application complexity grows, the difficulty of root cause analysis increases. Consider the difficulty of finding a root cause in a multi-tier application built on service-oriented architecture. The application may involve multiple services provided by multiple resources that may be scattered geographically and may reside both inside and outside the organization. In performing root cause analysis, developers and testers (in the development stage), and developers and support teams (in the maintenance stage) typically collaborate in a reiterative, cumbersome, three-step process that is often based on trial and error: > Gather problem information > Re-create the problem scenario in a controlled environment > Analyze the re-created problem to determine its root cause Many of the tasks are performed manually. As a result, the problem resolution process is inefficient and time consuming, driving up both test time (in the development stage) and support time (in the maintenance stage). 20% Writing Code 50% Problem Resolution 30% Other Figure 2. Developer time breakdown

Gather Problem Information When a problem is uncovered, the tester or support technician manually documents the problem and passes the information to the developer. To do so, he or she must manually collect the problem information from multiple, disparate sources, such as screen captures, internal application log files, operating system log files, memory dumps, and configuration information. The developer usually has to make several round trips to the tester (in the development stage) or to the support team or end users (in the maintenance stage) in order to gather the information necessary to ferret out the root cause. This process is time consuming, error prone, and communication intensive. It is important to note that while not all problems are difficult nor do they always take multiple iterations to determine root cause one of the key challenges of the problem resolution process is that developers and testers can rarely predict if a certain issue would be hard or easy to pinpoint; therefore, highly detailed information needs to be gathered for each issue. Re-create Problem Using the gathered information, the developer must re-create the problem scenario exactly, usually through a manual process. The developer typically begins by making assumptions as to why the problem is occurring and using these assumptions as a basis for recreating the problem. Accurate problem re-creation is extremely difficult because every problem can have literally hundreds of possible causes. In many cases, the developer s original assumptions are incorrect. As a result, the developer has to make several attempts at re-creating the problem before successfully replicating the original problem scenario. If the developer doesn t re-create the problem exactly, the same symptom may surface, but be due to a different root cause. Here is an example: A transaction failure caused by a program bug occurs in the QA system during testing. In attempting to re-create the problem in the development lab, the developer inadvertently misconfigures the application server. The incorrect configuration results in the same transaction failure and masks the real root cause, which is a program bug. While many problems are relatively easy to re-create, those that aren t consume huge amounts of time. In fact, some problems are deemed non-reproducible, because the developer cannot accurately re-create them. Analyze Root Cause Once the developer has re-created the problem, he or she must then perform an analysis to determine its root cause. Here again, the developer typically employs a manual, reiterative approach based on trial and error. The developer often uses a variety of tools and techniques in the analysis, employing different specialized tools for different categories of problems. The developer must first try to categorize the problem to determine which tools to use. For example, if the developer suspects memory leak, he or she may use a memory analyzer tool. Here again, a wrong assumption results in wasted time. Even if successful, the developer may have to re-create the problem multiple times in analyzing the root cause. For example, when using a debugger, the developer sets breakpoints at various locations in the source code, re-creates the problem, and steps through application execution. The developer then sets different breakpoints, re-creates the problem again, and steps through a different execution sequence, repeating this process until the root cause is uncovered. In summary, the application problem resolution process, which consists of information gathering, problem re-creation, and analysis, is manual. As a result, it is often error prone and inefficient. Optimizing this process would result in substantial efficiency gains to the development organization. Leveraging Technology to Streamline Application Problem Resolution New technology-based solutions are available to help application developers streamline and greatly accelerate the problem resolution process. Through the implementation of best-practice application problem resolution processes and the automation of manual tasks, these solutions can significantly reduce the time and effort developers now spend on resolving problems. Developers can leverage these solutions across the entire application lifecycle from development through maintenance. Streamlining the Development Stage A problem resolution solution can help the application development staff to streamline all problem resolution steps performed in testing in the development stage,. dramatically shrinking testing time. The solution should streamline the collaborative processes among developers and testers by automating information capture and facilitating information flow between the two groups.

Information Gathering The solution should also automate documentation of defects, providing a comprehensive, digital documentary of any problem encountered. In this way, it can improve tester productivity by automatically capturing and attaching comprehensive documentation of failures to a software defect report. It can also ensure that the developer receives, in the initial defect report, rich contextual information that facilitates root cause analysis. Because of the completeness of the information provided with the initial report, the developer does not have to go back multiple times to the tester for additional information, saving considerable time. Problem Re-creation Using the digital documentary that is created in the information-gathering phase and provided with the initial defect report, the developer is able to literally replay the problem scenario exactly as the tester experienced it. This is much like using a DVD to replay a movie. This replay capability eliminates the need for developers to manually re-create the problem scenario, saving a substantial portion of the developer s time. It also eliminates non-reproducible problems that soak up considerable time not only that of developers, but also that of testers and application support technicians. Information Analysis Because of the rich information provided by the digital documentary, the solution should also eliminate the need to manually categorize problems when performing problem analysis. This also minimizes the time needed to resolve the problem by eliminating the multiple problem analysis cycles typically needed to determine the root cause. Moreover, a well architected solution can replace the patchwork of disparate problem-analysis tools with a single, comprehensive, and integrated tool suite. This eliminates the need to make multiple passes through the problem with different tools, resulting in increased developer productivity, and enabling the developer to get to the root cause much more quickly. Streamlining the Maintenance Stage A problem resolution solution should also streamline the problem resolution process in the maintenance stage of the application lifecycle. Just as the solution can optimize the collaboration process between development and test engineers in the development stage, it can also optimize the collaboration process between developers (who are often the last line of support) and other groups involved in problem escalation in the maintenance stage (service desk, support team). The solution can streamline the incident documentation process by capturing a comprehensive digital documentary of incident occurrence in the production environment (including user behavior, the complete problem environment, and code execution). This ensures that all requisite information required to replay a problem scenario and analyze the root cause accompanies the initial problem report. The advantages include: > Eliminates the need for time-consuming, back-and-forth communications among the developer, the support team, and the end user to gather sufficient data > Eliminates non-reproducible problems that soak up considerable time on the part of many groups > Eliminates the need for developers to directly access the production environment (an industry best practice that helps organizations comply with regulations, such as Sarbanes-Oxley) > Makes the development organization substantially more efficient > Results in faster problem resolution and reduces. business risk Solution Criteria Because an application problem resolution solution is crucial to optimizing application development, it s important that the application development organization choose the right solution. This section presents a set of criteria that the staff can use in evaluating solutions. Support Entire Application Lifecycle The solution should support problem resolution processes across the entire application lifecycle, including both the development stage and the maintenance stage. This requires a solution that implements and automates best-practice processes that streamline collaboration between development and other groups involved in problem resolution, throughout the entire application lifecycle. These other groups include test engineers (in the development stage) and the application support team (in the maintenance stage). The solution should cater to the specific needs of each of the groups. For example, it should help testers easily document detected defects and enable developers to perform quick analysis of recorded information. Support Applications Across the Enterprise Look for a solution that can support applications across the enterprise. This requires the ability to capture data on all platforms, regardless of the platform operating environment (such as Microsoft.NET, IBM WebSphere, or BEA WebLogic), and regardless of where the platform is running (such as in a test lab, a production data center, or at a remote end-user site). The solution should also be independent of the development methodology used and the development environment employed.

Provide Thorough, Yet Unobtrusive, Data Capture It s important to look for a solution that can automatically capture complete problem data in a single pass to minimize re-creation of problems and facilitate root cause analysis of any type of application problem, including bugs in the source code, configuration issues, performance problems, user errors, and other issues. The solution must be capable of capturing information across all tiers in a multi-tier application environment. It should also capture virtually all application behavior at the source, including user interface, all interactions with external components (including database calls, Web Services, and operating system calls), the exact sequence of code-level execution, and the values of program parameters and variables. Data capture should be fast and easy to activate, and should not add excessive overhead to application execution. Most importantly, it should not require application change or disrupt business services in any way. Integrate Easily with Other Systems The solution should integrate easily with other systems that are currently used in application development and maintenance processes. This includes integration with systems that maintain problem data, such as defect tracking tools, testing tools, and homegrown logging mechanisms. It also includes integration with problem detection systems, such as IT infrastructure monitoring systems. In addition, this includes integration with systems that implement and automate processes relative to problem resolution, such as incident, problem, change, release, and configuration management systems. Compelling Business Benefits The implementation of an application problem resolution solution delivers several business benefits. Increase Business Agility A well-architected solution can help the application development staff save substantial time in problem resolution. As a result, the staff can deliver applications faster, without jeopardizing functionality or quality. This permits the development staff to increase business agility by responding more quickly to the needs of the business for new and enhanced applications. Reduce Development Costs Based on the experience of organizations that have already implemented application problem resolution solutions, a developer can save up to half the time spent in problem resolution. Using an assumption that developers spend half of their total time in problem resolution during development and maintenance, this translates to savings of more than one-fourth of the developer s total time per year. Assuming a fully loaded cost of $100 thousand per developer, a problem resolution solution would save $25 thousand annually per developer. For a team of 50 application developers, this translates into cost savings of $1.25 million per year. It can allow developers to focus more of their time on innovative projects that can make a company more competitive, instead of on problem resolution. Reduce Defects Because the solution greatly facilitates problem resolution, it permits more rigorous QA testing in the development stage, without extending the time to market (and often even shrinking it). This results in fewer defects in the production environment, which means less business risk. It also means less distraction for developers in handling problems in the maintenance stage of the application lifecycle. What s more, by eliminating non-reproducible bugs, the solution permits the development staff to remove a greater number of defects. Reduce Hiring Requirements Reducing the time developers have to spend on problem resolution substantially reduces staffing requirements for application development projects. With an automated application problem resolution solution in place, development managers can meet the functionality, quality, and time demands for new applications. Some Real-World Examples Many organizations have already begun to optimize application development using new problem resolution technologies and solutions. The results have been compelling. Here are just a few examples. Financial Services The private wealth management division of an international financial services provider has deployed more than 80 software applications to support its day-to-day operations. The majority of applications have been developed internally and tested in an offshore location. The company s development organization is required to release a new build of most customer-facing applications every four to eight weeks, and has found this objective to be a challenge, primarily due to developer attrition. The development organization addressed the challenge by implementing an application problem resolution solution that automated the problem resolution process. As a result, the division significantly improved the productivity of its developers and testers, reducing the developer hiring plan by more than 10 percent. At the same time, the solution

enabled the development organization to fix defects that were previously difficult to reproduce, increasing software quality without delaying delivery schedules. Healthcare Software Provider A leading provider of healthcare software solutions was developing a new healthcare system to improve patient care and management. The system would replace traditional paper charts with intelligent, interactive electronic forms. The company was eager to get the system to market, but in the healthcare industry, high quality is critical in software systems because people s lives are literally at stake. Using an application problem resolution solution, the software provider was able to shrink the time to market without jeopardizing quality. The company cut the time that developers had been spending on root cause analysis by about 80 percent. At the same time, the company was able to reduce the manual burden on its QA engineers by 25 percent, enabling that much more testing before release to production. Retail A large marketer of consumer electronics relies on 250 software applications to support and seamlessly integrate the operations of its three sales channels: retail stores, call center, and Web site operations. The applications, most of which have been developed in house, are tightly integrated and all share the same unified business processes on the back end. This ensures that customers have a consistent experience across all channels. As a result, customers get consistent access to special offers, pricing, and up-to-date inventory information, regardless of the channels they use. Conclusion By optimizing the problem resolution process through the implementation of an application problem resolution solution, the development organization can greatly increase application output in both quantity and quality. This enables development managers to meet the challenge of delivering more applications, in ever-decreasing time frames, and in a technology environment that is increasing in complexity all. without sacrificing functionality or quality. It also reduces hiring requirements, and that s welcome news to development managers. Today, application development is a key factor in business success. Application developers create the innovative business services that are revolutionizing business and are essential to business success. The more quickly and efficiently the application development staff can deliver applications, the greater the organization s business agility and resulting competitive edge. BMC offers solutions to optimize application development. For more information, visit www.identify.com. Endnotes 1 New Focus on IT Management, Ziff Davis Media, p.5, 2007 The applications development staff not only has to maintain this multitude of applications, but it must also enhance them and develop new applications to maintain the company s competitive edge. The staff implemented an application problem resolution solution to optimize application development processes and cut the time it takes to pinpoint and resolve problems in production. The solution enabled the company to reduce root cause analysis times by up to 70 percent, shrinking development cycles and improving service to application users through faster problem resolution.

About BMC Software BMC Software delivers the solutions IT needs to increase business value through better management of technology and IT processes. Our industryleading Business Service Management solutions help you reduce cost, lower risk of business disruption, and benefit from an IT infrastructure built to support business growth and flexibility. Only BMC provides best-practice IT processes, automated technology management, and award-winning BMC Atrium technologies that offer a shared view into how IT services support business priorities. Known for enterprise solutions that span mainframe, distributed systems, and end-user devices, BMC also delivers solutions that address the unique challenges of the midsized business. Founded in 1980, BMC has offices worldwide and fiscal 2007 revenues of $1.58 billion. Activate your business with the power of IT. www.bmc.com. About the Author Eldad Maniv, vice president of product management at BMC Software, is a leading expert in application development optimization. He has more than 20 years of experience in managing complex software projects in various industries. Since 2003, he has been collaborating with numerous organizations worldwide to analyze software development processes and build solutions for automation and optimization. Previously, he was founder and CEO at NextNine, a provider of problem resolution solutions for embedded systems. He also founded BiTec Ltd., a consultancy that delivered large-scale systems engineering projects in the telecom and medical industries. He holds a bachelor s degree in physics, mathematics, and computer science from the prestigious Talpiot program at Hebrew University in Jerusalem. To learn more about how BMC can help activate your business, visit www.bmc.com or call (800) 841-2031 BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. IBM is a registered trademark of International Business Machines Corporation. All other trademarks or registered trademarks are the property of their respective owners. 2007 BMC Software, Inc. All rights reserved. *71483*