lights-out testing for end-to-end business process validation
Contents Executive Summary...3 The Evolution of Testing...3 Lights-Out Testing Defined...4 Why Don t You Have It Already?...5 How Do You Get There?...6 What Is It Worth?...7 Preventing disruption to critical business processes...7 Accelerating delivery by reducing cycles time...8 Exploiting off-hour bandwidth for large, data centric integrated tests...8 Redirecting infrastructure and personnel to higher value tasks...8 Eliminating of manual validation and audit...9 Conclusion... 10 About Worksoft...11 2
Executive Summary Automated testing, especially for packaged applications such as SAP, has become a recommended best practice to ensure business processes are not disrupted by ongoing changes to the enterprise IT environment. By introducing stability into the production environment, end users avoid downtime associated with unavailable systems or functionality, and IT resources are less likely to be diverted into coping with crises and away from addressing the backlog of business needs. Both the business and IT can utilize time previously wasted on fire drills for more productive and effective activities. Additionally, the risks of inadequate or ineffective testing often go beyond strictly internal productivity measures to direct bottom line impacts. Issues with its ERP implementation reduced a leading US candy manufacturer s quarterly revenue by 25% during the industry s peak Halloween period, resulting in a revenue loss of $100 million. 1 Inadequate testing of a new discount fare rule cost a major international airline $50 million in badly needed revenue. 2 However, testing itself can be a costly and time consuming process. Manual or script-based testing requires involvement from experienced users, analysts or programmers whose time is scarce and valuable. Such labor-intensive testing also creates hard limits on the number of changes that can be processed at a time when updates and enhancements are happening at an increasingly more frequent rate. The solution to this dilemma and the next stage of the evolutionary cycle is Lights-Out testing, which reduces the need of human intervention by automating the testing process. Properly done, Lights-Out Testing yields these key benefits: Preventing disruption to critical business processes Accelerating delivery by reducing cycle time Exploiting off-hour bandwidth for large, data centric integrated tests Redirecting infrastructure and personnel to higher value tasks Eliminating manual validation and audit through automated documentation The Evolution of Testing Over the last decade, testing has evolved from a sporadic, manual activity to a partially automated process using scripted components to a project-based approach employing increasing levels of automation. But whether manual or automated, testing has consistently required manual intervention in some form, typically from experienced users or analysts whose time is scarce and valuable. During this same time period, the rate of change in software systems has steadily accelerated. Whereas in the past annual releases were the norm, these soon gave way to quarterly, then monthly, to weekly 1 CFO Magazine, September 2000 2 New York Times, September 12, 1988 3
and even daily changes. In fact, modern enterprises typically introduce hundreds of corrections, updates or enhancements on a constant basis. Further, these changes affect increasingly complex environments of interrelated applications both within and across enterprises. Unfortunately this rapid rate of change has outpaced the ability of traditional test approaches. Manual testing is no longer feasible given the intensive resource demands and extensive functional requirements. Scripted automation cannot keep pace either due to high development and maintenance overhead. In both cases, the need for some level of manual intervention introduces a level of friction in the overall delivery cycle that means many changes go untested, resulting in an unacceptable level of risk. By addressing these needs, a next-generation testing concept evolved that we call Lights-Out testing. This testing methodology builds on the lessons learned in traditional testing approaches and applies newer technological advances to create a streamlined and costeffective testing methodology. Lights-Out Testing Defined Lights-Out Testing is defined as recurring, automated and unattended validation of essential business processes. In this context, each of these terms has a particular meaning: Recurring means that testing is applied to every change, whether such changes occur daily, weekly or at any other interval. This requires that tests can be repeated as often as needed without time-consuming setup or maintenance. This has special implications for test design, data management and implementation, as explained below. Automated means text execution is rapid, accurate, and consistent with a fully documented audit trail. Only automation can enable a reasonable level of coverage within highly compressed delivery cycles and restricted resource availability, yet it must do so without injecting additional complexity and overhead. This dictates an evolutionary advance in test automation technology. Unattended means the tests can be performed at any hour of any day or night with no human intervention required. This capability is indispensable to the concept of Lights-Out : In many cases testing must occur overnight, after all changes have been made and before they are introduced to production. But to be effective, unattended testing must be resilient enough to cope with unexpected conditions without either aborting the entire test cycle or generating spurious downstream errors. ê ê Validation means confirming the accuracy and availability of end-to-end business processes. Unlike traditional tests that center on discrete functionality in departmental silos, endto-end validation traces the outcome of business rules and flow of data across platforms, applications and operational areas. This approach has the highest probability of exposing the unexpected impact of changes to integrated systems, but it requires an advanced test and data design methodology. A well designed validation approach can yield detailed documentation that may reduce time, effort and costs for compliance and regulatory audits. 4
ê Essential means the business processes that power critical corporate operations. In other words, Lights-Out Testing is about reducing or removing the most risk in the least amount of time, not about verifying each and every detailed requirement. This laser focus is necessary given the tight time frames that are a reality of incessant change. More expansive testing can and should be performed at planned intervals based on larger scale projects such as upgrades, new deployments, mergers or acquisitions, but Lights-Out Testing is all about making sure the business can continue its business the next day. Why Don t You Have It Already? As previously discussed, traditional testing approaches are inadequate in today s environment. Manual testing is no longer practical because it takes too long and costs too much to perform. Only expert users or analysts have the requisite business process knowledge and understanding to design and execute meaningful tests, and their time is both scarce and expensive. Time spent testing is time not spent on streamlining and improving corporate operations to increase profitability. But even if these resources could be made available, there is simply not enough time. With changes often occurring on a daily basis, manual execution is too slow, inconsistent and error prone to be effective. Further, tests performed by individuals tend to be specific to that person s area of expertise, so that endto-end validation is rarely done. As a result, coverage is inadequate. Traditional automation hasn t worked either. Despite advances in component design, scripting language-based tests take too long and cost too much to develop and maintain. Specialized technical skills are required to develop the components, thus excluding the subject matter experts and introducing extra steps and expense. The resulting script code is fragile and breaks too easily when unexpected conditions are encountered. 5
Even after the initial investment has been made in code components, the maintenance effort is too timeconsuming to keep up with rapid change. Coded components must be examined for impact, then modified, tested and debugged before they are reliable enough for execution. The bottom line is that writing code to test code is inherently problematic: Code is the problem, not the solution. The obvious conclusion is that you need a quantum shift in your test approach to deal with the new reality. That shift is Lights-Out Testing. How Do You Get There? Lights-Out Testing is recurring, automated and unattended validation of essential business processes that serves as an accelerant to the daily flow of changes across the enterprise IT landscape. But for most companies that are struggling just to keep their head above the rising tide of business demands, technology advances and functionality explosion, the concept of Lights-Out Testing may seem desirable but so far distant as to be unrealistic. The good news is that you can start from wherever you are and arrive at Lights-Out Testing by adopting the right set of attitudes, people, process and technology. Adopt New Attitudes embrace change as an undeniable and even desirable aspect of surviving in a competitive, agile marketplace. Devote yourself to the idea that testing can be a lubricant to the change process by reducing the risk of production outages or errors that require emergency responses and drain resources from meeting the needs of the business. People Are Critical Your human resources are the most critical aspect of effective testing. You have to know which business processes are vital, what business rules are in play and what data is consumed or produced. Since few companies have current and complete business process documentation readily available, this means you have to extract this knowledge from those very scarce resources. However, once complete, maintaining documentation for Lights-Out Testing is far less burdensome than for manual or script-based testing. Process Is Important In order to execute a set of business processes on a regular even daily basis, you must plan the execution to simulate a day in the life of production. That is, the processes must be ordered around dependencies so that the operating cycles are preserved. ê ê Technology is Key To succeed at Lights-Out Testing, you must adopt a completely new paradigm that allows you to automate testing by creating data instead of code. The benefit of using a data model for Lights-Out Testing is the ability to leverage the power of a relational database to simplify the development and maintenance of test processes. By maintaining relationships among all aspects of your test assets, a change to any element can be instantly traced to each related item, providing the capability to instantly analyze the impact of changes on all affected test assets and then automatically update them. 6
What Is It Worth? Lights-Out Testing ensures that critical business processes are not disrupted by ongoing changes to the enterprise IT environment. Implementing such a test program involves added time and costs, so the question is whether it is worth it. Can such a program generate a positive return on investment (ROI) within a reasonable time frame? Properly done, Lights-Out Testing yields these key benefits: Preventing disruption to critical business processes An ounce of prevention is both the most valuable benefit and the least quantifiable. It s difficult to measure the cost of problems prevented, but this doesn t make the savings any less real. Errors exposed to customers may impact the value of the brand; errors exposed to suppliers can cripple the supply chain. Downtime reduces user productivity and responding to errors or failures reduces IT capacity. Any or all of these can affect enterprise agility and profitability. It is less difficult, of course, to measure the cost of problems not prevented. Consider the impact of the failure of a major bank to process any financial transactions for ten full days after a routine change of its brand name. 1 As previously discussed, issues with its ERP implementation resulted in a revenue loss of $100 million for a major candy manufacturer, and inadequate testing cost a major international airline $50 million. These are just a few examples as the list goes on. Beyond these immediate effects is the cost of diverting time and resources towards correcting application and update errors. While most businesses do not track these costs directly, they are reflected in lower productivity and profitability, higher backlogs, and away from meeting business and IT goals. 1 New York Times, June 7, 2004 7
Accelerating delivery by reducing cycles time A streamlined and comprehensive testing process translates directly into improved business agility. Lights-Out Testing can convert daily changes from a game of Russian roulette to a competitive weapon that results in greater market share, customer loyalty and bottom line profitability. The fact is that traditional test approaches are so slow and cumbersome that they act as speed bumps to change, forcing updates to be either planned with military precision and executed at the speed of dark or crammed down daily without any testing at all or only spot checking based on fears and hunches. For example, a national insurance company found that its roll-out of a new dental coverage plan was being slowed significantly due to the testing required for the unique regulatory requirements in each of all 50 states and 6 territories. Their regulatory environment dictated that not testing was not an option. In contrast, rapid, predictable test cycles that fit smoothly into the change cycle on a routine basis give the business the confidence and stability it needs to focus on its customers and competitors instead of its infrastructure. That same insurance company was able to vault from a pace of one state every four months to 32 states in the same period, dramatically accelerating their market penetration and revenue potential. Exploiting off-hour bandwidth for large, data centric integrated tests Running test tasks during non-critical hours reduces the downtime or freeze frame typically associated with testing during which changes must come to a screeching halt while testing occurs. It also opens access to a wider array of environments and power that are not usually dedicated to testing but are available during off hours. This capability expansion empowers the test organization to achieve greater coverage and therefore reliability without a corresponding increase in costs. For example, a global process manufacturing organization was able to validate 250 critical end-to-end processes in a single, Lights-Out session. This dramatic improvement in speed and efficiency was achieved with a decrease in the time and costs that previously yielded with much lower test coverage. Redirecting infrastructure and personnel to higher value tasks Finding the best and highest value use for your resources can increase productivity without increasing costs. To that end, Lights-Out Testing has two important implications. Infrastructure can be exploited during off-hours, as explained above, and personnel who would otherwise be diverted into tedious and time-consuming manual testing can refocus their efforts to streamlining and improving business processes and operations. In one case, an aircraft fuselage manufacturer was able to save 8 man months of testing after only their first project. 8
But a less obvious yet more valuable benefit ensues from the avoidance of out of window emergency changes that are necessitated by errors or failures. By introducing stability into the production environment, end users avoid downtime associated with unavailable systems or functionality and IT resources are less likely to be diverted into coping with crises and away from addressing the backlog of business needs. Both the business and IT can utilize time previously wasted on fire drills for more productive and effective activities. That same aircraft manufacturer realized fewer emergency transports and was able to increase attention to long-awaited improvements. Eliminating of manual validation and audit This may be one of the most easily measured benefits, simply because many companies dedicate valuable resources to validating internal controls and then also pay outside auditors to validate their compliance. Simple costing methods can quantify the displacement of these expenses achieved through an automated approach. In one case, a Worksoft customer saved over $250,000 per year every year in outside audit fees by providing detailed documentation created as a by-product of automated validation. What is less obvious is the opportunity cost of performing or paying for these tasks and the opportunity benefit of making them available for other purposes. In the above example, the extra $250,000 per year was now available to be invested elsewhere with a potentially higher return. Similarly, the human resources previously consumed by manual validation are now available to pursue more promising projects. 9
Conclusion Given the nature of today s business environment and the demands on IT to ensure the reliability and productivity of enterprise systems, implementing a Lights-Out Testing solution will reduce risk, increase productivity and control IT costs. While the actual costs associated with inadequate testing, as well as the benefits achieved through Lights-Out Testing, are unique to each company and situation, the underlying factors are the same. Traditional testing methods provide only a partial solution and do not address the root issues. The key to capping your organization s test data management costs and risks is to implement the fully automated, recurring and unattended validation of essential business processes that Lights-Out Testing provides. Organizations can no longer ignore the escalating costs associated with deploying inadequately testing application updates and changes. This is why hundreds of companies rely on Worksoft products to implement test data management solutions for their enterprise applications such as SAP. Proven in enterprise installations worldwide, Worksoft Certify is a revolutionary change to testing methodologies, providing a Lights-Out Testing infrastructure that meets business goals by reducing test cycle times, balancing bandwidth, ensuring that critical business processes perform without disruption, increase staff and resource productivity and automate documentation. Together, Worksoft and your IT organization can uncover the risks and costs you may be experiencing today and help predict and measure the benefits that Lights-Out Testing can make a reality in your world. 10
About Worksoft Worksoft is next-generation test automation that brings speed and agility to SAP and the eco system of software that surrounds it by automating end-to-end testing, not just for SAP, but for everything SAP interacts with even web interfaces. Unlike legacy test automation systems, Worksoft is easy to use, allowing Worksoft customers achieve 60-70% automation in 90 days enabling lights out testing even for minor transports and releases. There s a reason why over 100 of the largest SAP customers in the world use Worksoft for test automation. Discover Worksoft today. Get Agile in your SAP Ecosystem. For more information, contact Worksoft at 1-866-836-1773 or visit www.worksoft.com. 2012 Worksoft Inc. All rights reserved. Worksoft and Worksoft Certify are registered. All other trademarks are properties of their respective owners. 11