TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING Validation and Testing: A CA Process Map Malcolm Ryder CA SERVICES / ARCHITECT MARCH 2010
Executive Summary Challenge Table of Contents Executive Summary 1 SECTION 1: Finding the Right Path to IT Excellence 3 SECTION 2: Validation and Testing 5 Adopt Best Practices Validate & Tracking Validations Versus Perform Tests Report / Closure SECTION 3: Benefits 9 A Key to Achieving IT Excellence SECTION 4: Conclusions 10 SECTION 5: About the Author 10 The Information Technology Infrastructure Library version 3 (ITIL V3) process framework approaches IT (ITSM) in terms of the lifecycle of a service. The Lifecycle is an organization model providing insight into the way ITSM is structured, and embodies critical guidance for IT organizations seeking to improve service quality and align more closely with business goals to create value for the business and its customers. However, ITIL V3 best practice guidelines across the five phases of the service lifecycle are complex and challenging to interpret, and they are not designed to provide definitive advice about implementing ITSM processes. Many IT organizations consequently undertake an ITIL journey without a firm idea of their goals and the path to achieve those goals. One of the key ITIL management processes, Validation and Testing, presents its own special challenges. Participants in the service quality assurance effort must coordinate the priorities and extent of testing with the type and timing of business needs to be met; consequently, the measure of success in validation and testing extends beyond the technical aspect to the matter of business value. Opportunity The primary objective of the Validation and Testing process is to ensure that a service offering, as designed, deployed and supported, has the quality expected to meet customer expectations.this quality is a strategic process outcome. CA has developed a unique approach to representing the ITIL framework and its interdependent IT (ITSM) processes at a high level in the form of an easy-to-use subway map. This map is an ideal starting point for understanding and communicating about ITIL in support of successful program planning and implementation. Benefits Following the Validation and Testing map provides guidance to: Improve business process capabilities Effectively upgrade the IT infrastructure in timely alignment with business needs Foster the efficient delivery of multiple IT services Improve the quality of technical support Copyright 2010 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies. This document is for your informational purposes only. To the extent permitted by applicable law, CA provides this document As Is without warranty of any kind, including, without limitation, any implied warranties of merchantability or fitness for a particular purpose, or non-infringement. In no event will CA be liable for any loss or damage, direct or indirect, from the use of this document including, without limitation, lost profits, business interruption, goodwill or lost data, even if CA is expressly advised of such damages. TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING 1
Info. Security Capacity Assess & Classify Assets Maintain Policy Analyze Performance Build Plan Issues Meet Report Achievements Catalog Availability s Build Specify Methods/ Continuity Techniques Set Security s Forecast Model/Trend Document Definition Portfolio Assess Risk Determine Vulnerabilities Build Catalog Analyze Contents Test Proactive Revise SLAs / OLAs IT Continuity Review Views Technical Views Publish Live s Catalog Operational s Customer Satisfaction Test Adjust and Tune Performance Maintain s Security Incidents Review and Audit Mitigate Risk Optimize Availability Design SLA Framework Catalog Level IT Continuity Capacity Info. Security Availability Deliver Required Resources Level Continual Validation and Testing Asset and Transition Planning and Support Release and Deployment Release and Deployment RFC Planning Transition Planning and Support Prioritize Continual Develop Strategy and Planning Evaluation/ Decision Transfer Categorize Impact CAB Review Execute (Emergency/ Standard) Build and Report Coordinate Resources Maintain Accurate s Responsiveness Status Reports Identify s and Plan Audit Portfolio Build/Release Assure Quality Asset and Status Reports Report/ Closure Perform Tests Deploy Adopt Best Practices Validate & Ensure Release Value Validation and Testing Request Fulfillment Problem Incident Event Request Fulfillment Access Access Request (Incl. Self-) Provide Rights Access Request Continual Secure Access Prevent and Eliminate Problems / Track Informed Decisions Resolve Record Executive Policy Transfer/ Disseminate Problem Deliver Standardized s Error Portfolio Review/ Action Transform to Usable Store Info. Known Errors Automate and Capture Info. Approval (, Compliance) Fulfillment Resolve/ Recover Restore Raise Incident Filter / Detect Correlate (Incl. Fault Detection) Select Response Record Investigate Diagnose Escalate Incident Work Around Event Problem Simplifying ITIL The ITIL V3 process framework focuses on the service lifecycle and the way that service management components are structured and linked. It embodies critical guidance for IT organizations that are seeking to improve service quality and align more closely with business goals. But, the ITIL V3 best-practice guidelines across the five phases of the service lifecycle are complex and challenging to interpret. Moreover, they are not designed to provide definitive advice about implementing IT (ITSM) processes. Many IT organizations consequently undertake an ITIL journey without a firm idea of their goals and the path to achieve those goals. SECTION 1: CHALLENGE FIGURE A CA has developed three maps: Design, Transition and Operation since most ITSM discussions focus on these critical ITIL disciplines Finding the Right Path to IT Excellence Design Transition Operation CA has developed a unique approach to charting the ITIL journey through a visual representation of the ITIL framework and its interdependent ITSM processes modeled after an urban subway system. This three-part map (Figure A) presents an easy-to-navigate, high-level view of the ITIL terrain. IT executives, strategists and implementers can use these process maps along with the family of CA process map technology briefs that expand on them. The maps and technology briefs provide a common reference point for understanding and communicating about ITIL and help you with program planning and implementation. FIGURE B The Transition map represents a journey of developing and improving capabilities for the transition of new and modified services to production. Transition Release and Deployment Planning Transition Planning and Support Develop Strategy Evaluation/ Decision Impact CAB Review Execute (Emergency/ Standard) Maintain Accurate s Responsiveness Audit Portfolio Adopt Best Practices Validate & Ensure Release Value Validation and Testing How to Use the CA Process Maps CA s process maps (Figure A) illustrate every process (or track), each activity (or station) and the key relationships that are relevant to navigating continuous IT service improvement. The ITIL quality cycle takes the form of a circle with each Plan-Do- Check-Act (P-D-C-A) step as a process integration point (junction) on the line. Junctions serve both as reference points when assessing process maturity, and as a means to consider the implications of implementing a process in isolation. Validation and Testing Asset and Transition Planning and Support Release and Deployment RFC Prioritize and Planning Continual Transfer Categorize Build and Report Coordinate Resources Build/Release Identify s and Plan Assure Quality Asset and Report/ Closure Perform Tests Deploy Strategic controls ( Portfolio, and ) are needed to reduce risk and optimize integration across the service lifecycle, as illustrated on the three points of the triangle centered in the P-D-C-A quality circle (seen more easily in Figure B). These strategic controls help in evaluating, prioritizing and assuring the appropriate levels of financial and human resources for existing and new services. This paper is part of a series of Process Map technology briefs. Each brief explains how to navigate a particular ITIL process journey, reviewing each process activity that must be addressed in order to achieve process objectives. Along each journey careful attention is paid to how technology plays a critical role in both integrating ITIL processes and automating ITIL process activities. Most organizations are highly sensitive to quality levels in IT deployments, and they already rely on continuously managed processes and practices that are expected to provide some assurance of the fitness of deployed components and systems for their purpose. For that matter, validation most often means verifying that component and systems specifications are appropriate to established norms for operational risk. Testing takes place under the watchful eye of domain and subject matter experts in the line of duty for resolving problems and executing changes. Yet as we see in the process interactions shown on our subway map, the circuit from planned modifications to deployment releases is an incomplete journey. This is because builds and installations may or may not go according to expectations. But once utilization begins, quality requirements differ again, and acceptance of the deployments and releases depends on further steps defined more from a business user s perspective than from a technical one. 2 TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING 3
Alignment and compliance with the business is the overall goal of validation and testing, mandating explicit awareness of business requirements and coordination with them. Key points in this coordination include: Cooperation of specific business and technical roles Documentation of requirements at different levels of service, and corresponding documentation of the plan for validations Tracking the validations versus the requirements Another question that comes up is about how much testing is enough. Too much is difficult to justify, and too little is too risky. One of the main outcomes of the Validation and Testing process is an increase in the ability to recognize a return on the investment in testing, which encourages the inclusion of right-sized effective testing in the business-level planning and budgeting for IT s development. SECTION 2: OPPORTUNITY Validation and Testing The Validation and Testing process is responsible for planning and coordinating tests to ensure that specifications for the service design are met and validated through delivery of the service and, including co-operation with the Release and Deployment process, to manage and limit risks that could result from insufficient utility and warranty of the service in operation. Validation and Testing takes the technical proficiencies and plans of quality assurance for IT and strategically places them in the larger context of service quality for the business. The strategic issue is that the business must be able to grow and adapt in order to maintain its success and critical competencies in its market ecosystem of customers, partners and competitors. This makes both Capacity and strategic areas of risk, and consequently it makes investments in service deliveries and service modifications part of financial management as well as business risk management. The ultimate goal of the Validation and Testing track (Figure C) is a full alignment of characteristics to the utility and warranty requirements of the business. The subway map for this journey plots the key stations along the course in this way: Adopt Best Practices Validate and Perform Tests Closure and Reporting FIGURE C The Validation and Testing track represents management s attention to how verification of service characteristics is proactively aligned with the requirements for utilization and warranty of an upcoming new deployment of service 4 TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING 5
Adopt Best Practices The key practices of Validation and Testing will be A model for identifying business alignment An integrated set of test plans Training, knowledge transfer, and communications Verification of results through pre-defined tracking and reporting These practices are represented in the following table, which illustrates the incorporation of testing within a larger process framework (inputs, activities, and eventual outcomes) of identifying business value from quality assurance. The inputs for this high-level process express two things: a proactive identification of the customer s view of a service, and the perspectives of the testing organization as driven by that view. Explicit definitions and standardized formulations of the relevant factors across the process are reusable, which allows for direct comparisons of different testing occasions. The results of such comparisons create an awareness that supports prioritization of service deliveries; and they inform analyses for the improvement of quality management procedures. QUALITY ASSURANCE Inputs Validations Outcomes Quality Policy Value Expectations Package Description of Designed IT Delivered Value Available and Supportable Tracking Validations Versus Specs decompose from top down business to service to supplier Tests build from bottom up components to functions to business agreements SPECIFICATIONS VALIDATION TESTS KEY ROLES Alignment to Agreements Customer Compliance r User Acceptance Level r Test r Test r User Solution Design Operational Readiness r Release Release Package Security r Release Manger Development Components with Assembly Technical and Subject Matter Experts Level Package Intended Utility and Warranty Utility Level (fit for purpose) Warranty Level (fit for use) Release for Targeted Line of Qualified Supplier(s) and Targets Measures Test Strategy Plans, Model Resources Methods, Scripts Usability Compliance Certification Specifications Buisness Needs Quality Rating and Results Report In the table above, items such as Level Packages (SLPs), Test Plans, and Reports are typical artifacts of the ITIL management process and can be researched in detail within most substantive ITIL publications. A service management organization will want to reach a level of confidence that the various items exist and are clearly related to each other. This will call for assigning clear responsibilities for routine production and review of the items. Validate & The outstanding feature of Validation and Testing is its deliberate synchronization of test types with defined business requirements, agnostic to particular technologies and to the variations of hardware and software. The following illustration presents an overview of this synchronization. Perform Tests The essence of manageable testing is the ability to define tests as repeatable, measurable procedures. This introduces the aspect of Key Performance Indicators (KPIs). As noted in most literature on performance management, KPIs are metrics that address efficiency, effectiveness and cost issues traceable to the design of an executing process and related to the outputs of that process. Indicators identify output characteristics that correlate to desired outcomes, and the means of driving those indicators will be seen as Critical Success Factors (CSFs). Accordingly, testing will feature standards, scripts (methods) and metrics that are defined and logically related to each other before test execution actually begins. Further, as a matter of setting scope to testing as well as methodological compliance, policies will be established to govern the decisions about when testing must occur and what type must occur. As identified in the table above, key roles in the organization take on responsibility and authority as stakeholders in the specifications to be validated. In that light, cooperation amongst the roles is expected and directed by policies, while also being institutionalized as integrations of their respective management processes. At points of process integration, the processes may feed each other as participants in shared workflow. In the ITIL subway map of processes, this is notable especially where the Build and Release of a service package involves validation and testing. 6 TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING 7
One of the most important aspects of these process integration points is the documentation that occurs and is used from validation and testing. For example, a test plan usually precedes release and deployment activities, and an update of service records in a configuration management database (CMDB) usually follows deployment but precedes a postimplementation review (PIR). This arrangement highlights the cooperation expected between managers of services, configurations, changes and releases. Report / Closure Documentation of results is also a fundamental deliverable of the Validation and Testing effort, as testing itself is expected to be the source of evidence that the service is not ready for delivery and may go through another iteration of development or modification before it can succeed. As part of proper management of testing itself, these cycles of tests need directives identifying and justifying whether the scope and depth of testing is to be different or not in the next cycle versus the prior one. Because test cycling directly affects the turnaround time between development and release, the management of the test iterations affects the ability of the organization to meet service level agreements with customers. Test reporting explicitly establishes the reasons why something is tested and why it is deemed suitable for release or not. As material for decision support, the reporting affects resource allocations as well as readiness for compliance auditing. SECTION 3: BENEFITS ITIL v3 describes Design, Transition (deployment), and Operation as three major phases in the service lifecycle. Within Transition, the Validation and Testing effort is one of several high-level management processes that link the appropriate design of the service to the smooth operability of the service. IT s are the building blocks of business processes and business services. A successful delivery of a service will feature the right thing arriving at the right place at the right time for the right reason. Validation and Testing confirms that the service is fit to its target purpose and fit to its offered use. The business uses multiple services to support its own performance and growth. Executives own the service portfolio, set service goals and control the investment in the service. This executive demand and influence thus should sponsor the qualification of the service as a primary concern of financial and risk management for the business. The executive view of a successful service implementation includes concerns about the control, compatibility and delivery of the service. These three factors translate, respectively, into management of the quality, risk, and value of the provided service, as deliberately generated from end-to-end throughout the service lifecycle. The goal of Validation and Testing is to ensure that specifications for the service design are met and validated through delivery of the service and, including co-operation with the Release and Deployment process, to manage and limit risks that could result from insufficient utility and warranty of the service in operation. A Key to Achieving IT Excellence Automating ITSM through technology can help your organization reduce the amount ofresources required to achieve ITIL v3 best practices. This assists your IT department in improving the quality of its services while embarking upon a continuous IT service excellence program focused on fostering business growth. As you reach the end point of the Validation and Testing journey outlined in the CA Transition process map, your organization should have a better handle on the steps needed to promote successful service deployments. Specifically, bringing deployment efforts in line with ITIL best practices can help you: Improve business process capabilities Effectively upgrade the IT infrastructure in timely alignment with business needs Foster the efficient delivery of multiple IT services Improve the quality of technical support 8 TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING 9
SECTION 4: CONCLUSION Validation and Testing facilitates ITSM by managing the certification of IT in the form of services needed by the business. However, doing so requires rigorous processes, appropriate communication plans and support commitments as well as an outline detailing where to start and how to proceed through the journey. Following the steps outlined in the CA Transition process map gives organizations a clear view of how their Validation and Testing journey will take shape, illustrating the key stops en route to achieving effective requirements alignment. This journey results in: Strategic re-orientation of testing towards business value Better prioritization and investment in services Improved risk management Stronger integration with other ITSM and ITIL best practices About CA: CA (NASDAQ: CA) is the world s leading independent IT management software company. With CA s Enterprise IT (EITM) vision and expertise, organizations can more effectively govern, manage and secure IT to optimize business performance and sustain competitive advantage. For more information, visit www.ca.com. SECTION 5: ABOUT THE AUTHOR Malcolm Ryder has over 25 years experience in the IT industry, with expertise in the areas of service transition, IT value management, and IT strategy. For the last 15 years, Malcolm has worked in consulting and solution strategy roles with a heavy emphasis on service management systems with vendors, service providers and end-user customers. Malcolm has been a co-developer of multiple market-leading commercial ITSM solutions since the mid 80s. This document shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. The development, release and timing of any features or functionality described in this document remain at CA s sole discretion. Notwithstanding anything in this document to the contrary, upon the general availability of any future CA product release referenced in this document, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such releases may be made available to current licensees of the product who are active subscribers to CA maintenance and support, on a when and if-available basis. In the event of a conflict between the terms of this paragraph and any other information contained in this document, the terms of this paragraph shall govern. Copyright 2010 CA. All rights reserved. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Intel and Intel Itanium are trademarks or registered trademarks of Intel Corporation in the United States and other countries. Microsoft, SQL Server.Net, Visual Studio, Internet Explorer, Windows Vista and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. ITIL is a Registered Trade Mark, and a Registered Community Trade Mark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. Red Hat and Red Hat Enterprise Linux are trademarks or registered trademarks of Red Hat, Inc in the United States and/or other countries. SUSE is a registered trademark of Novell, Inc., in the United States and other countries. Eclipse is a trademark of the Eclipse Foundation, Inc. SPARC is a trademark or registered trademark of SPARC International, Inc. in the United States and other countries. Adobe and Dreamweaver are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries. Mozilla and Firefox are registered trademarks of The Mozilla Foundation in the United States and/or other countries. Solaris is a register trademark of Sun Microsystems in the United States and other countries. IBM, z/os, z/architecture, CICS, WebSphere. DB2, MQSeries, System p5, System p6, AIX and DB2 Universal Database are trademarks or registered trademarks of International Machines Corporation in the United States, other countries, or both. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies. THIS CUMENT IS FOR YOUR INFORMATIONAL PURPOSES ONLY. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS CUMENT AS IS WITHOUT WARRANTY OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT WILL CA BE LIABLE FOR ANY LOSS OR DAMAGE, DIRECT OR INDIRECT, FROM THE USE OF THIS CUMENT, INCLUDING, WITHOUT LIMITATION, LOST PROFITS, LOST INVESTMENT, BUSINESS INTERRUPTION, GOODWILL OR LOST DATA, EVEN IF CA IS EXPRESSLY ADVISED IN ADVANCE OF THE POSSIBILITY OF SUCH DAMAGES. ID1688-032910 10 TECHNOLOGY BRIEF: SERVICE VALIDATION AND TESTING