SAP Standard for Root Cause Analysis

Transcription

1 SAP Standard for E2E Solution Operations Document Version: SAP Solution Manager 7.1

2 Typographic Conventions Type Style Example Description Words or characters quoted from the screen. These include field names, screen titles, pushbuttons labels, menu names, menu paths, and menu options. Textual cross-references to other documents. Example EXAMPLE Example Example <Example> Emphasized words or expressions. Technical names of system objects. These include report names, program names, transaction codes, table names, and key concepts of a programming language when they are surrounded by body text, for example, SELECT and INCLUDE. Output on the screen. This includes file and directory names and their paths, messages, names of variables and parameters, source text, and names of installation, upgrade and database tools. Exact user entry. These are words or characters that you enter in the system exactly as they appear in the documentation. Variable user entry. Angle brackets indicate that you replace these words and characters with appropriate entries to make entries in the system. E X A MPLE Keys on the keyboard, for example, F2 or ENTER SAP SE or an SAP affiliate company. All rights reserved. Typographic Conventions

3 Document History Version Date Change First version created Document History 2014 SAP SE or an SAP affiliate company. All rights reserved. 3

4 Table of Contents 1 SAP Standards for E2E Solution Operations Control Center Approach Overview of the Standard for Root Cause Analysis Lifecycle of Root Cause Analysis Plan Technical Prerequisites Organizational Prerequisites Build Run Optimize Driving Continuous Improvement Quality Assurance Tasks Quality Targets and KPIs Training Technical Expert Competence Workshops More Information SAP SE or an SAP affiliate company. All rights reserved. Table of Contents

5 1 SAP Standards for E2E Solution Operations IT organizations face new challenges every day as they attempt to remain effective and future safe while also keeping costs for day-to-day operations as low as possible. They are also being challenged more than ever to demonstrate their value to businesses. Therefore, it is important to optimize the day-to-day tasks that have less obvious business value and to use KPI and benchmark-based reporting to make IT processes more visible, demonstrating the real value that IT can provide. In order to minimize the costs of IT, it is necessary to standardize and automate IT processes end-to-end (E2E) without reducing the SLAs required by the business, such as stability, availability, performance, process and data transparency, data consistency, IT process compliance, and so on. Based on the experience gained by SAP Active Global Support (AGS) while serving more than 36,000 customers, SAP has defined process standards and best practices to help customers set up and run E2E solution operations for their SAP-centric solutions. The Build phase of SAP best practices supports a Build SAP Like a Factory approach, consisting of the following processes: Custom code management Change, test, and release management Incident, problem, and request management Solution documentation Remote supportability During the Run phase of a solution, adapting your IT infrastructure to a Run SAP Like a Factory operation impacts both application operations and business process operations. Therefore, operations processes, such as technical monitoring, end-to-end root-cause analysis, technical administration, and data volume management need to be optimized to achieve state-of-the-art application operations. In business process operations, the same applies to business process and interface monitoring (including performance optimization), data consistency management, and job scheduling management. Quality management processes and tasks need to be established throughout the lifecycle to guarantee continuous improvement of the end-to-end operations processes while simultaneously ensuring the flexibility needed to react to changing requirements. SAP Standards for E2E Solution Operations 2014 SAP SE or an SAP affiliate company. All rights reserved. 5

6 Figure 1: Organizational Model for Solution Operations This figure shows an organizational model for solution operations that aligns SAP best practice topics and E2E standards with SAP's control center approach. The Operations Control Center executes and controls the Run SAP Like a Factory processes, while the Innovation Control Center ensures optimal custom code management and a smooth transition to production with integration validation procedures. SAP connects to these control centers from the Mission Control Center to ensure that professional support is available to the customer. The following Application Lifecycle Management (ALM) functions are not provided directly in one of the control centers because they must be handled across different areas: Change, test, and release management Incident, problem, and request management Solution documentation Remote supportability The quality management methodologies are an essential part of SAP's Advanced Customer Center of Expertise (CoE) concept and ensure that the KPI-driven processes are continuously improved across all processes and teams. In addition, the quality manager roles ensure consistent and value-centric reporting to the business and management. This unified reporting platform is known as the Single Source of Truth. 1.1 Control Center Approach The Operations Control Center (OCC) is the physical manifestation of the Run SAP Like a Factory philosophy. The OCC allows for automated, proactive operations, which simultaneously reduces operational costs while increasing the quality of IT services, leading to improved business satisfaction. The OCC also drives continuous improvement of business processes and IT support. To achieve these goals, it relies on a close interaction with both the Innovation Control Center (ICC) and the SAP Mission Control Center (MCC) SAP SE or an SAP affiliate company. All rights reserved. SAP Standards for E2E Solution Operations

7 Figure 2: Interaction Between ICC, OCC, and MCC The OCC is a central IT support entity at the customer site, which monitors the productive SAP environment as well as important non-sap applications. During operation, the OCC requires a workforce of 2 full-time equivalent (FTE) per shift to ensure that incidents are detected and resolved as quickly as possible. The OCC is equipped with large screens that display the status of business processes, IT landscape components, as well as exceptions and alerts. If problems occur, you use a video link to get live support from SAP and partners. The customer usually sets up the room with assistance from SAP Active Global Support (AGS). The customer is responsible for managing the OCC and the team of technical and functional IT operators who act on the alerts. The OCC is most effective when closely integrated with other IT processes, such as IT Service Management (ITSM) and Change Management. Central monitors and dashboards based on application and business process operations display the current status of business and IT-related processes. This data can also be used to drive continuous improvement. An effective system monitoring and alerting infrastructure is fundamental to the success of an OCC. SAP Standards for E2E Solution Operations 2014 SAP SE or an SAP affiliate company. All rights reserved. 7

8 Figure 3: OCC Concept The OCC is most effective when closely integrated with other IT processes, such as IT Service Management (ITSM) and Change Management. Central monitors and dashboards based on application and business process operations display the current status of business and IT-related processes. This data can also be used to drive continuous improvement. An effective system monitoring and alerting infrastructure is fundamental to the success of an OCC. For Job Scheduling Management, the OCC supervises all background monitoring processes, SAP controls and legacy background operations. It reacts to job monitoring alerts according to predefined error-resolution activities, and triggers follow-up activities for error handling if the relevant task are not completed within a certain timeframe SAP SE or an SAP affiliate company. All rights reserved. SAP Standards for E2E Solution Operations

9 2 Overview of the Standard for Root Cause Analysis In today s service-enabled, distributed, heterogeneous system landscapes based on a variety of technologies, such as ABAP, Java, and.net, business processes can span multiple system components within the IT landscape. End users can now also access these systems from a variety of mobile devices using multiple channels. Identifying the root cause of incidents is increasingly challenging and requires a systematic, top-down approach to isolate the erroneous component and subsequently resolve the issue. The E2E root cause analysis process and associated E2E tools are usually controlled by the application support team. When service is interrupted, the end user creates an incident in the IT Service Management (ITSM) component of SAP Solution Manager. The incident creation procedure requests information, such as urgency of an incident, a short description, categorization, and so on. If there is no previously identified workaround for the incident, you can use the E2E root cause analysis tools provided by SAP and partners to identify the cause and begin solving the problem. E2E root cause analysis can also be triggered by an alert in the various monitoring functionalities in SAP Solution Manager, such as technical monitoring or business process and interface monitoring. The Monitoring and Alerting Infrastructure (MAI) integrated into in SAP Solution Manager evaluates events and, depending on the configuration of technical monitoring scenarios, triggers an alert, which automatically creates an incident and starts the workflow for incident resolution or problem management. Figure 4: Triggers for E2E Root Cause Analysis Overview of the Standard for Root Cause Analysis 2014 SAP SE or an SAP affiliate company. All rights reserved. 9

10 A problem manager is then responsible for assigning resources to the problem. The assigned resources can use the E2E root cause analysis tools and roadmaps to perform a top-down E2E analysis across multiple components. The E2E root cause analysis standard mainly consists of the requirements that should be fulfilled in order to support both customers and SAP support consultants during the problem resolution process. This standard also describes the basic process for a project-based implementation of E2E root cause analysis with SAP Solution Manager. Figure 5: High-Level Process Flow SAP SE or an SAP affiliate company. All rights reserved. Overview of the Standard for Root Cause Analysis

11 3 Lifecycle of Root Cause Analysis 3.1 Plan This chapter explains the prerequisites and steps necessary to implement E2E root cause analysis. Within the planning phase, it is important to consider both the technical and organizational aspects of the implementation and make sure that all prerequisites are fulfilled Technical Prerequisites To implement E2E root cause analysis, the following technical prerequisites must be fulfilled. SAP Solution Manager 7.1 with the latest support package stack is installed and operational. Before you install a new version of SAP Solution Manager, you need to make sure you have sufficient hardware resources to operate SAP Solution Manager and the technical components required for root cause analysis. For more information about the required hardware sizing, see the Sizing Toolkit available from SAP Service Marketplace at Application Operations End-to-End Root Cause Analysis Media Library. A central system landscape directory (SLD) is configured and up-to-date. Although you configure a dedicated SLD for SAP Solution Manager as part of SAP Solution Manager Configuration, a central SLD containing all SAP systems must be configured and contain up-to-date system information (software components, patch levels, hostnames, and so on) before you start setting up the diagnostics infrastructure. Wily Introscope Enterprise Manager is installed on the SAP Solution Manager host or on a separate host. The diagnostics infrastructure of SAP Solution Manager includes technology from CA Wily Technology, namely Wily Introscope, which is mainly used for SAP AS Java analysis. CA Wily Introscope is shipped with a preconfigured set of dashboards on a right-to-view license, which covers the tool's main diagnostic functionalities for all supported SAP products. CA Wily Introscope uses byte code instrumentation (BCI) technology to collect and integrate performance statistics at code level for Java and.net components, without having to access the source code. For more information, see the installation guide for Wily Introscope on SAP Service Marketplace at Application Operations End-to-End Root Cause Analysis Media Library Introscope Setup Guide. SAP Solution Manager diagnostics agents are installed. As a general rule, one diagnostics agent must be installed for each virtual host of each managed system. The diagnostics data provisioning layer does not rely on a monolithic agent. Instead, it is built on a two-level agent architecture. This allows diagnostics agent applications, such as Wily Introscope agent, to be deployed from SAP Solution Manager during the Managed System Configuration step or while updating SAP Solution Manager. For more information about setting up diagnostics agents, see the following SAP Notes: o SAP Note Lifecycle of Root Cause Analysis 2014 SAP SE or an SAP affiliate company. All rights reserved. 11

12 o SAP Note o SAP Note If you use SAP Solution Manager with connected managed systems, make sure that the Self-Diagnosis functionality in the System Monitoring work center does not report errors for the Diagnostics and RCA Managed Systems function groups. 1. In the System Monitoring work center, open the Self-Diagnosis view. 2. Choose Self-Diagnosis Settings. 3. On the Functions tab page, deselect the function groups Diagnostics and RCA Managed Systems. 4. Confirm your selection Organizational Prerequisites To implement E2E root cause analysis, the following organizational prerequisites must be fulfilled: Personnel have received adequate training to be able to perform E2E root cause analysis. Personnel are assigned the necessary authorizations in SAP Solution Manager and in the managed systems. The process is designed in alignment with Incident Management, Problem Management, Change Request Management, Test Management, Knowledge Management, and Continuous Improvement. The requirements for integrating non-sap root cause analysis tools are fulfilled. Resources are assigned to implement and configure SAP Solution Manager and the managed systems in the analysis scope. 3.2 Build You have the following options for configuring E2E root cause analysis in SAP Solution Manager: Read the relevant documentation and execute the necessary steps by yourself. Book an Expert Guided Implementation (EGI). During the session you will learn about the basic configuration of SAP Solution Manager and managed systems. By the end of the session, SAP Solution Manager is ready to use for root cause analysis. For more information, see SAP Service Marketplace at Book an SAP support consultant to visit you on-site and perform the configuration together with your staff. To request this service, contact Consulting, your dedicated customer contact, or your local Support Advisory Center. Basic Overview of the Configuration Process 1. Start SAP Solution Manager Configuration (transaction SOLMAN_SETUP). 2. Perform all activities in the Basic Configuration scenario. On screen help is available for each step of the guided procedure. 3. Perform the Managed System Configuration scenario for each system. E2E root cause analysis is now available for each system you configured SAP SE or an SAP affiliate company. All rights reserved. Lifecycle of Root Cause Analysis

13 Recommendation To check that the infrastructure is working properly, SAP recommends that you configure the Self- Monitoring functionality in the Technical Monitoring scenario. For more information, see the documentation on SAP Library at Run SAP root cause analysis compromises tools for client and server-side analysis. The most important tool is the Root Cause Analysis (RCA) work center in SAP Solution Manager. For both SAP and the customer, the RCA work center is the central point of access to all functions for analyzing errors and fulfills the requirements of the SAP E2E standard for root cause analysis. The navigation concept of the RCA work center follows a top-down analysis approach. The analysis usually starts in the End-to-End Analysis view, where you identify the component responsible for the actual problem. You can then perform a deeper analysis on the identified component using the views for system, host, and database analysis. You can access additional tools directly from the Related Links section. The main screen area displays all SAP systems connected to SAP Solution Manager and provides access to the specific analysis tools. The End-to-End Analysis view contains the following tools: Exception Analysis provides centralized analysis of all exceptions that occur in the managed systems. This includes ABAP dumps and ABAP syslog errors, as well as Java application errors collected from the default trace, and specific log and dump views. Workload Analysis aggregates-server-side performance statistics of managed systems to identify general server-side performance bottlenecks, such as sizing issues. If you are experiencing performance issues, the E2E Workload Analysis tool can help you to identify the problem. Change Analysis tracks all changes (technical configuration, code, content) that are applied to the managed systems. This information is especially useful if a few ad-hoc changes cause disruption in a productive system; instead of drilling down through the landscape, you can identify the problem by comparing different systems, which is usually quicker and easier. Trace Analysis isolates a single user request in a complete landscape, providing trace information on each of the system components. The measurement starts at the end user s interface (Internet Explorer or SAP GUI). With the help of an individual correlation ID, each request is traced throughout the SAP server landscape. This function enables the customer to quickly identify the component causing the problem. The RCA work center also contains views for the following analyses: System Analysis contains the Change Reporting and Log Viewer tools. Host Analysis provides access to predefined file system folders (File System Browser) and enables SAP Support to execute certain read-only commands using the OS Command Console. Database Analysis provides access to different database statistics, which are based on data collected by the DBA Cockpit. Lifecycle of Root Cause Analysis 2014 SAP SE or an SAP affiliate company. All rights reserved. 13

14 3.4 Optimize The E2E root cause analysis tools help to simplify the problem resolution process. SAP Solution Manager Self- Monitoring, in conjunction with SAP Solution Manager Self-Diagnostics, provides you with tools to view the functional status of SAP Solution Manager and related components. However, these tools are only accurate if you keep SAP Solution Manager and the related technical infrastructure components up-to-date. You need to update the SAP Central Correction Note (CCN) on a regular basis using the Note Assistant (transaction SNOTE) or the Implement SAP Note step in the System Preparation scenario of SAP Solution Manager Configuration (transaction SOLMAN_SETUP). Applying corrective actions usually involves a change, which is triggered by the relevant component expert. Changes to productive systems have to be tracked and approved by change request management. The technical system owner (TSO) initializes any supplementary follow-up activities in coordination with the IT strategist. The TSO is the person in charge of a specific SAP system (including the database) and is responsible for applying major changes to that system and ensuring professional system documentation is maintained. The result of the RCA process is stored in the customer's solution database. The component expert is responsible for the documentation process. If several experts have contributed to the solution, this process is primarily led by the TSO. You should document any workarounds that helped restore productive operations so that you can resolve any new occurrence of the same issue quickly by referring to the documentation. Ideally, you should make proactive changes to avoid similar situations in the future, for example, by changing or adding alert thresholds within the technical monitoring scenarios. To provide optimal transparency regarding incidents and problems, use the Solution Documentation functionality to document your business processes and application operations across your entire landscape. SAP recommends implementing an Advanced Customer Center of Expertise (CoE) to verify your IT organization according to SAP best practices, tools, and processes SAP SE or an SAP affiliate company. All rights reserved. Lifecycle of Root Cause Analysis

15 4 Driving Continuous Improvement It is important that your organization learns from each incident or problem so that your root cause analyses and subsequent problem resolution processes become increasingly efficient and effective. This section describes some key steps you can take to drive continuous improvement for E2E root cause analysis. 4.1 Quality Assurance Tasks From a quality management perspective, the key tasks are as follows: Define clear ownership of the issue Ensure a message escalation process is in place and monitored Ensure activities to prevent incidents are identified and deployed (monitoring, automation, and so on) Monitor documentation quality and follow-up RCA KPIs Track the incident backlog and message resolution times to identify focus areas for follow-up with RCA Track resolution rates and time taken to identify the root cause for each expert level Ensure all RCA tools related to SAP Solution Manager are implemented, for example, diagnostics tools Set up a dedicated RCA team that is trained and has the experience to holistically analyze critical issues 4.2 Quality Targets and KPIs To ensure continuous improvement of the root cause analysis process and drive the value recognition of IT, the most important quality targets are as follows: Improve stability and continuity of business processes and reduce business risks Speed up problem resolution process by simplification within an IT environment Increase efficiency by reducing the total cost of ownership To assess the quality of the root cause analysis process, you need clearly defined parameters and measurable objectives. The key parameters should be collected and evaluated in regular reports. The historical data that is created in this way can be used to identify trends and then derive the necessary measures to take. The following table describes the main challenges for each of these quality targets and which KPIs can be used to measure them: Driving Continuous Improvement 2014 SAP SE or an SAP affiliate company. All rights reserved. 15

16 Quality Targets Challenges KPIs Improve stability and continuity of business processes and reduce business risks The more often your systems are down and the longer it takes to resolve the issue, the greater the impact on your business. Availability percentage of business hours Average time to resolve priority 1 issues Speed up problem resolution process by simplification within an IT environment Speed up the process of identifying the root cause and to create a structured approach to resolution. A structured approach is important because it ensures that the focus of the discussion remains on resolving the issue Number of components investigated before the issue was resolved Number of components routed more than once To avoid unnecessary work and reduce the time for root cause identification, you need a KPI to measure the following: Number and percentage of issues for which the root cause is identified Average time to identify the root cause for priority 1 and priority 2 issues Increase efficiency by reducing the total cost of ownership Ensure quality while reducing costs and increasing speed. This can be achieved by replacing costly, timeconsuming, individual RCA processes with standard, structured, and tool-supported root cause analysis By implementing standardized root cause analysis, resolution times, the number of resources involved in the investigation, and the required level of expertise are reduced significantly Average resolution rate for each expert level RCA performed with tool support according to defined standards Number of root causes identified by customer vs. root causes identified by partners To take full advantage of root cause analysis, you need to perform follow-up actions on root cause identification and proactively track the solution. It is also important to monitor incident messages and resolution times to identify focus areas for improvement as early as possible. Number of reopened problems If root cause analysis is set up correctly, this number should be close to zero. Track the incident backlog Track message resolution times SAP SE or an SAP affiliate company. All rights reserved. Driving Continuous Improvement

17 5 Training The E2E Solution Operations Curriculum helps customers to train their specialized teams and stakeholders efficiently in various operations topics. Individuals learn about SAP standards, which describe best practices, correct use of tools, and collaboration between different roles. The E2E Solution Operations Curriculum is structured based on different knowledge levels. E2E050 E2E Solution Scope and Documentation This course explains the solution concept implemented by SAP Solution Manager and outlines the solution documentation process for SAP-centric scenarios. This course is aimed at teams and stakeholders involved in E2E Solution Operations. E2E040 Run SAP End-to-End Solution Operations This course is aimed at customer IT management and project managers. It provides a general introduction to the Run SAP methodology, as well as a detailed explanation of the different standards for planning and adopting Run SAP. The course also describes the success factors critical to E2E Solution Operations. This course is aimed at teams and stakeholders involved in E2E Solution Operations. Technical Core Competence courses These courses explain important day-to-day system administration tasks in detail. These courses consist of demos and exercises aimed at application management, SAP technical operations, business process operations, and custom development. E2E100 End-to-End Root Cause Analysis E2E100 covers tools and approaches for analyzing the root cause of incidents and isolating the component causing the incident in a heterogeneous IT landscape. Prerequisites for this course include an overview of SAP technology as well as basic understanding of SAP Solution Manager. E2E100 is a five-day course ending with a certification exam in the afternoon of the fifth day. Upon successful completion of this certification, the participant receives the certificate Application Management Expert Root Cause Analysis. This certificate proves that the candidate has a basic understanding within this profile and can implement this knowledge practically in E2E solution operations. Course Content: Introduction to End to End Root Cause Analysis Incident Management Process and Pre-clarification End to End Change Diagnostics End to End Workload Analysis End to End Trace Analysis End to End Exception Analysis Data Inconsistency Analysis Client Side Root Cause Analysis Proving Stabilization of Solution Landscapes with SAP EWA Training 2014 SAP SE or an SAP affiliate company. All rights reserved. 17

18 Review and Certification Preparation Certification for the Application Management Engineer End to End Root Cause Analysis 5.1 Technical Expert Competence Workshops These courses are individual, on-site customer workshops performed using the customer s own systems and held by a component-specific expert from SAP. During the session, an expert explains basic and specialized tasks regarding solution operation of certain SAP systems, such as SAP NetWeaver Portal, SAP NetWeaver Process Integration (SAP NetWeaver PI), SAP NetWeaver Business Warehouse (SAP NetWeaver BW), or SAP Customer Relationship Management (SAP CRM). The course is targeted at members of application management, for example, IT component experts and technical system owners. For a detailed description, see SAP Service Marketplace at Expert Competence System Administration. You can also book supplementary workshops about BMC AppSight and Wily Introscope SAP SE or an SAP affiliate company. All rights reserved. Training

19 6 More Information Documentation Monitoring and Operations with SAP Solution Manager IT Service Management with SAP Solution Manager SAP Service Market Place SAP Wiki Link SAP PRESS published by Galileo Press SAP PRESS published by Galileo Press More Information 2014 SAP SE or an SAP affiliate company. All rights reserved. 19

20 SAP SE or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies. Please see for additional trademark information and notices. Material Number: