Best Practice Test Management for Business Suite on HANA Migration Projects using SAP Solution Manager 7.1

Transcription

1 Best Practice Test Management for Business Suite on HANA Migration Projects using SAP Solution Manager 7.1 November 2013 Marcus Wefers, Chief Support Architect, ALM Solution Management, AGS, SAP AG Sebastian Geissler, Senior ALM Consultant, SAP Deutschland AG & Co. KG

2 Agenda: Test Management for Suite on HANA migration projects Introduction and Project Phases Phase 1: Test Preparation Business Blueprint Test Scope Identification Functional Regression Tests Performance and Load Tests Service and Interface Tests Phase 2: Test Execution Phase 3: Test Activities after Go-Live Key Take Aways 2013 SAP AG. All rights reserved. 2

3 Introduction and Project Phases

4 Test Management for Suite on HANA migration projects HANA migration project phases and involvement of test management Suite on HANA Migration Project Preparation phase Suite on HANA migration project after go-live Test Management Status assessment Tool & approach decisons Test management setup Test planning Test execution Issue resolution Retest and sign-off Regression tests for HANA revision deployments 2013 SAP AG. All rights reserved. 4

5 Phase 1: Test Preparation

6 Test Management for Suite on HANA migration projects Test Management status assessment and setup activities (1) Assessment & Decision Setup Workstream Business Blueprint Coverage of business processes documented Identification of Business Analysts Interviews with business teams to identify prio 1 business processes Business processes and involved executables RBPD service or SAP SolMan: Solution Documentation Assistant or SAP SolMan: Scope and Effort Analyzer (Q1 2014) Workstream Test Scope Identification Metrics for test scope identification - examples: priority or criticality of business process # of affected users usage frequency of business process Optional: metric assignment to business processes (ABC classification) Functional Test Plan 1 Test case selection from business processes with class A metrics Functional Test Plan 2 test case selection from business processes including custom code / modifications via ABAP Test Cockpit + UPL SQL Monitor Workstream Coverage assessment of manual and automated tests Optional: Implementation BPCA Functional Target level for test automation, e.g. 30% project for selected test Test case definition Regression Tests Test tool evaluation and tool selection +. ROI expectations automation tool selected functional tests for execution by QA teams E2E integration tests for execution by Business Analysts Workstream Test Planning Status test system (PRD copy, test data, ) Availability of business users for tests Reporting requirements Test system setup Definition of Test Packages from Test Plans Tester assignment 2013 SAP AG. All rights reserved. 6

7 Test Management for Suite on HANA migration projects Test Management status assessment and setup activities (2) Assessment & Decision Setup Metric selection for load profile definition Performance Baseline Definition of Load Profile scenarios examples: Workstream Performance Tests Load Tests Workstream Service & Interface Tests Top (15 ) transactions, batch jobs, reports creating 80% of system load performance critical transactions Throughput of critical transactions peak hours and maximum number of users Transactions with high number of affected users Processes with custom code or modifications Identification of non-assigned systems Identification of SAP and non-sap systems required, but not connected to HANA test system Tool selection for Service (system) Virtualization Identification of critical Services and APIs Identification of (web) services and APIs which require tests without UI (e.g. w/o calling App) and multiple use cases / data variants Tool selection for service testing define baseline for comparisons Workload Monitor (ST03N) SAP Workload (STAD) Performance Analysis (ST05) Definition of scaling factor for PRD versus TST system using SAPS ratio Service (system) Virtualization high number of online users performing critical transactions in parallel throughput of processes (with 3rd party interfaces) critical processes including CC / modifications Creation of Performance Test Scripts Optional: DB Stress Tests: test programs and queries to perform high volume DB access and postings SAP recommendation: connect required system if possible and avoid system virtualization Optional: virtualization of not connected, but required systems using system virtualization SW Service Testing Creation of service test cases to perform automated tests for (web-) services or APIs without application. Data variation to perform various regression tests of the service / API under test. Workstream Reporting & Sign-off Definition of Reporting Requirements Decision: use of add. reporting features Selection of reports / optional: setup of add.reporting Definition of sign-off criteria 2013 SAP AG. All rights reserved. 7

8 Phase 1: Test Preparation 1a - Business Blueprint

9 SAP Solution Manager Solution Documentation of System Landscape and Business Processes System landscape documentation Business process documentation DEV TST PRD SAP CRM TST DEV PRD SAP ERP Single source of truth Non-SAP business apps Infrastructure 2013 SAP AG. All rights reserved. 9

10 Business Blueprint E2E Business Processes and automated generation of SAP Module oriented Blueprint Business Blueprint with E2E business processes Sales Order to Cash Business Blueprint with process steps by SAP Modules LE Logistic Execution LE-SHP Shipping Quotation Delivery (VL01N) Sales Order Delivery Goods Issue Billing E2E business process Goods Issue (VL02N) SD Sales & Distribution SD-BIL Billing Billing (VF01) Process steps grouped by SAP Modules Procurement SD-SLS Sales Procure to Pay Alternatives to create E2E Business Blueprint 1. File upload using existing process documentation 2. Solution Documentation Assistant (tool) and Reverse Business Process Documentation (service) 3. Data transfer from ARIS or HP QC (tool by ALM COE EMEA) 4. Manual documentation supported by BPR (SAP content) Quotation (VA21) Sales Order (VA01) Automated generation of SAP Module oriented Blueprint 1. Application: Scope and Effort Analyzer Availability: SAP Solution Manager 7.1 SP11 (Q1 2014) Blueprint items generated in background based on usage statistics of executables in PRD systems Grouping by SAP Application Component Hierarchy (ACH) 2013 SAP AG. All rights reserved. 10

11 Business Blueprint Creation of Business Blueprint using Solution Document Assistant of SAP Solution Manager 7.1 SAP Solution Manager Content Interface RBPD Content Projects 3 4 Solution 1 Solution Documentation Assistant 1 - Upload business process structure and rules 2 - Analyze managed systems 3 - Create and complete project (custom objects) 4 - Hand over to solution 2 SAP ERP SAP CRM SAP SolMan Other SAP Solution Landscape 2013 SAP AG. All rights reserved. 11

12 Business Blueprint Automated generation of SAP-Module oriented Blueprint - Result Result 3 level Blueprint hierarchy Level 1: SAP Module Level 2: ACH L2 Level 3: Transaction or Report Tab Transaction with assigned executable Custom Code Node Customer Custom Code transactions and reports are grouped by development class 2013 SAP AG. All rights reserved. 12

13 Business Blueprint Composition of E2E Business Processes based on SAP-Module oriented Process Step Library From Process Step Library. Automatic generation to E2E Business Processes Manual composition of E2E business process 2013 SAP AG. All rights reserved. 13

14 Phase 1: Test Preparation 1b - Test Scope Identification

15 Test Scope Identification Functional Test Plan #1: Metrics used for identification of important business transactions Step 1: Identification of metrics Prioritiy / criticality of business process Number of affected users of business process Usage frequency of business process Step 2: BPx guidance ABC classification of business process priorities through interviews with Business process Experts (BPx) of all business areas Step 3: Usage frequency analysis Assess the usage frequency with SAP tools like Workload Monitor (ST03N) ABC classification for top n business transactions Step 4: ABC classification for Bus. Blueprint Definition of custom attributes for Business Blueprint in SAP Solution Manager from step 1 Assign attribute value for business processes with A classification (from step 2 and 3), e.g. high priority / criticality high number of users high usage frequency 2013 SAP AG. All rights reserved. 15

16 Test Scope Identification Functional Test Plan #1 for important business transactions Test Option 1: SAP Solution Manager with integrated partner products Test Option 2: SAP Quality Center by HP plus SAP TAO Business Blueprint Manual and automated Tests Manual Test Business Blueprint SAP Solman Adapter: transfer processes with Class A classification Automated Test QC Requirements Module Test Plan generation - alternatives: 1. filter all available test cases by Blueprint process attribute 2. filter Blueprint by keywords which are assigned to test cases QC Test Plan Module Manual tests HP QTP and SAP TAO tests QC Test Set generation Functional Test Plan #1 Functional Test Plan # SAP AG. All rights reserved. 16

17 Optimize custom code and modifications Motivation Main drivers for optimizing custom code for the Suite on HANA migration: Discover the potential and take the opportunity to drastically improve the performance of long running transactions using HANA functionality Restore the performance of transactions due to performance impacts of HANA migration Fix functional degradation due the HANA migration There is no need to adapt everything! Identify what needs to be done and where improvement potential is available 2013 SAP AG. All rights reserved. 17

18 Test Scope Identification Functional Test Plan #2 for Custom Code Approach to identify Custom Code with potential performance improvements ABAP Test Cockpit (ATC) Usage and Procedure Logging (UPL) SQL Monitor (SQL-M) find violating SQL statements with static code analysis fix functional & performance degradation by adaption of ABAP/OpenSQL Statements, like SELECT for Pool/Cluster-Tables w/o ORDER BY Native SQL code (Critical Statements) DB hints (Critical Statements) ATC results: functional prio 1 issues UPL logs all called and executed ABAP units like programs, function modules down to classes, methods and subroutines ATC results + UPL: exclude unused custom code from analysis limit analysis to high usage areas Dynamic runtime analysis of SQL statements Aggregated logging of all ABAPbased SQL executions Ability to identify custom defined tables with high access frequency Performance issues CC tables Narrow down ATC results SAP recommendation: define Regression Test Plan #2 for adjusted code objects 2013 SAP AG. All rights reserved. 18

21 Test Scope Identification Functional Test Plan #2 for custom code: Usage and Procedure Logging (UPL) Usage and Procedure Logging (UPL) 1 Execute business transaction UPL is a kernel based logging technology providing runtime usage information of ABAP procedure units like methods, function modules, subroutines and much more SAP Business Suite System like SAP ERP 2 Load ABAP procedure units UPL complements the standard ST03N workload statistics of ABAP executables (system role: PRD) SAP Kernel UPL provides 100 % reliable usage analysis without measurable performance impact 3 Log ABAP usage UPL UPL is available as of SAP Netweaver 7.01 SP10 with Kernel 720 Patch 94 SAP Solution Manager 4 Evaluate usage 2013 SAP AG. All rights reserved. 21

23 Test Scope Identification Functional Test Plan #2 for custom code: customer example (modified) Customer example (modified) The company activated UPL well in advance to the Suite on SAP HANA migration project > 1000 used Custom Code objects detected by UPL ABC classification for CC usage (3 months) Class A: > 100 mio executions Class B: > 1 mio executions ATC code scans results limited to Custom Code of usage frequency class A and B 1 2 ABC classification of CC usage frequency via UPL CC adjustments for prio 1 ATC results limited to Class A+B ATC scan pattern Findings Fixed SELECT for Pool/Cluster-Tables w/o ORDER BY >10 >10 Native SQL code (Critical Statements) >10 >10 DB hints (Critical Statements) <10 <5 Analysis of WHERE Condition for SELECT 1-10 <5 Analysis of WHERE Condition in UPDATE and DELETE 0 0 Changing Database Accesses in Loops >100 <20 EXIT or no statement in SELECT...ENDSELECT loop 1 0 SELECT Statements with Subsequent CHECK 0 0 Incomplete evaluation. <x>% of fields used <10 0 SELECT Statements That Bypass the Table Buffer >10 <10 Table Attributes Check 0 0 SAP recommendation: define Regression Test Plan #2 for adjusted code objects 2013 SAP AG. All rights reserved. 23

24 Test Scope Identification Functional Test Plan 2 for custom code: SQL Monitor SQL Monitor # executions # records selected table name Entry point High # executions Zero # records High # records High mean time 2013 SAP AG. All rights reserved. 24

25 Test Scope Identification Functional Test Plan #2 for Custom Code Approach to identify Custom Code with potential performance improvements ABAP Test Cockpit (ATC) SQL Monitor (SQL-M) SQL Performance Tuning Worklist Combines ATC and SQL-M in one application Transaction code: SWLT Availability with SAP Basis 7.02 SP14 / 7.38 SP09 / 7.40 SP SAP AG. All rights reserved. 25

27 Test Cope Identification Functional Test Plan 2 for custom code: SQL Performance Tuning Worklist (T-Code: SWLT) SQL Performance Tuning Worklist correlates Static Code Checks covering golden rules from ABAP Test Cockpit Runtime information (numbers of execution, execution time, fetch count, ) from SQL Monitor Create a weighted work list of code occurrences that need attention Provide solution proposals 2013 SAP AG. All rights reserved. 27

28 Test Cope Identification Example for prioritization and resolution using SQL Monitor SQL Performance Tuning Worklist provides information about Runtime information (from SQL Monitor) Static Checks from ABAP Test Cockpit (ATC) and includes starting point of executable, URL, RFC, (e.g. transaction VA01) detailed information on how to improve the ABAP statement 2013 SAP AG. All rights reserved. 28

29 Test Scope Identification Functional Test Plan 2 for custom code: Business Process Change Analyzer (BPCA) Goal Optimal performance after HANA migration No negative impact on critical business processes in combination with effort optimized regression test scope SAP Application ABAP Test Cockpit (ATC) Usage and Procedure Logging (UPL) SQL Monitor SQL Performance Tuning Worklist (SWLT) Business Process Change Analyzer (BPCA) - SAP Solution Manager SAP and partner test tools Activity Identification of Custom Code and Modifications with problematic ABAP statements Adjust code based on priority and usage Identification of business processes impacted by CC adjustments Test scope optimization Regression tests of impacted processes 2013 SAP AG. All rights reserved. 29

30 Test Scope Identification Functional Test Plan 2 for custom code: Business Process Change Analyzer (BPCA) 1 Business Blueprint 2 TBOM (*) Tests (**) Custom Code and Modification adjustments Transport adjusted Custom Code / Modifications 3 BPCA Change impact analysis selection of adjusted custom code / modifications from transports Selection of business processes, TBOMs, test cases BPCA change impact analysis plus test scope optimization 4 Test Plan generation (semi-automatic) in SAP Solution Manager or SAP Quality Center (*) Semi-dynamic TBOMs generation fully automated with SAP SolMan 7.1 SP10 (**) Test cases for HP QC integrated with Business Blueprint and BPCA via SAP-HP interfaces 2013 SAP AG. All rights reserved. 30

31 Phase 1: Test Preparation 1c - Functional Regression Tests

32 Test Management Products and capabilities integrated with SAP Solution Manager 7.1 New SAP Solution SAP Solution Update Business Blueprint Change Impact Analysis Test Planning Test System Setup Test Execution Change Deployment Business Blueprint Test Option 1 Test Management Test Workbench Test Automation Test Automation Framework CBTA HP QTP 3 rd Party BPCA Test Option 2 Test Management Quality Center Test Automation Business Process Testing SAP TAO HP QTP Test Option 3 SAP Connector, Quality Manager, Requirements Composer, Team Concert, SAP AG. All rights reserved. 32

33 Phase 1: Test Preparation 1c - Functional Regression Tests Test Ooption1 using SAP Solution Manager 7.1

34 Test Management Test Option 1 using SAP Solution Manager 7.1 Capabilities of Test Option 1 with SAP Solution Manager 7.1 (SP07 and beyond) Business Blueprint Change Impact Analysis Test Planning Test System Setup Test Execution and Reporting Change Deployment SAP Solution Manager 7.1 Test Case Assignment Business Blueprint Test Scope Optimization Test Planning WoC Test Management Test Plan generation Tester assignment Test Execution WoC Test Management Manual Tests Automated Tests Test Status Reporting WoC Test Managment Test Workbench BW Reporting Partner Logs and Reports Business Process Change Analyzer Lean test system setup SAP TDMS (BPCA) Test Automation Framework Test script creation Test system assignment Test data assignment CBTA (Component based Test Automation) ecatt HP QTP WS Certify other 3rd party test automation tools 2013 SAP AG. All rights reserved. 34

35 Manual Testing Manual Test Case for business transaction with description of execution steps and access to test data Note: manual test case is assigned to process step in Business Blueprint via T-Code SOLAR SAP AG. All rights reserved. 35

36 Manual Testing Convinient access for manual Testers Tester Worklist 1) 1 Easy access for manual testers from Work Center Test Management 2) 2 Test Packages assigned to Tester including status 3) 3 Test Cases of selected Test Package - Start of execution - Test status - Test Notes - Incidents SAP AG. All rights reserved. 36

38 Manual Testing Test Execution: user views test data to be used during test execution Second window with view of available test data of TDC stays open during test execution 2013 SAP AG. All rights reserved. 38

39 Manual Testing Test Execution: guided access to System Under Test (SUT) to start execution of business transaction Screen 1: test script Screen 2: test data Test execution Screen 3: automatic launch of transaction that shall be tested by user in preselected test system 2013 SAP AG. All rights reserved. 39

40 Test Automation with SAP Solution Manager 7.1 Test Automation Framework Non-SAP Test Design Test Execution Test Result Analysis Accelerated Repair Seamless integration between SAP Solution Manager and test tools from SAP and Partners Start of automated tests from Workcenter Test Management Test Manager can view test results from manual and automated tests in Workflow between Tester and Test Engineer to trigger test case repair Test Data Container (TDC) provide test data for test script parameters Central management of System under Tests Ad-hoc start or scheduler for unattended execution Integrated test execution logs from partner tools one environment Status and progress resports Integrated partner reports Rich environment for Test Engineer to analyze problems and perform repair activities 2013 SAP AG. All rights reserved. 40

41 Test Automation Framework in SAP Solution Manager 7.1 Automated tests assigned to process steps in Business Blueprint Test Configuration Business Process Test Script Test Data System under Test CBTA, ecatt Order Type Customer Product OR C1000 P-100 HP QTP OR C2000 P-200 other ISV test automation tools Test Data Container Component Based Test Automation (CBTA) is shipped with SAP Solution Manager 7.1 SP07 Customers with SAP Enterprise Support contract can download 2 HP QTP licenses from SMP ( Test Automation Framework provides 2 interfaces to integrate ISV test automation tools. SAP ICC offers certification services SAP customers have a choice of test automation tools integrated with SAP Solution Manager 7.1 through the Test Automation Framework 2013 SAP AG. All rights reserved. 41

42 Component-based Test Automation (CBTA) Test Automation of E2E Business Processes using SAP Solution Manager - CBTA Business Process Order to Cash Quotation Sales Order Delivery Goods Issue Billing Component-based Test Automation (CBTA) Test Composition Environment (TCE) Business Analyst executes business transaction via CBTA CBTA creates test script based on test components during execution Attributes Parameters Script Components Test Data Business Transaction Systems under Test Parameters for test data inputs Dynamic handover between scripts SAP delivered test components On the fly generated components Data defined in Test Data Container Test Data Assignment Wizard 2013 SAP AG. All rights reserved. 42

43 Automated Tests Flow to create new automated test using Component-based Test Automation (CBTA) Business Process Test Composition Environment (TCE) create new Test Configuration Component Based Test Automation (CBTA) CBTA Test Creation Wizard Test Composition Environment (TCE) finalize Test Configuration Quotation Sales Order Delivery CBTA process flow analysis Store Test Configuration at process step or business process CBTA Test Components / Test Script Test Data Assignment Wizard assign test data Test Data Container 2013 SAP AG. All rights reserved. 43

44 Functional Regression Tests Test data planning 1 Test Data Container (TDC): Data Structure 2 TDC: test data planning test data entry test data upload How: create TDC parameter for all input fields of your business process used in test script Who: Test Engineer Who: Business Analysts Use: Test data stored in a TDC can be consumed by multiple Test Configurations. Test Data Container(s) act as central repository for your test data. Customer benefit: test data changes can be done in one central location leading to a significant lower maintenance effort and faster availability of test data 2013 SAP AG. All rights reserved. 44

45 Functional Regression Tests Test execution of automated tests Tester Worklist in Workcenter Test Management 5 Test results and logs Test Script SAP CBTA, HP QTP, 1 User starts test execution Test Configuration 3 Test execution: handover of Test Script + Test Data + SUT info 4 Logon and automated business process execuction Test Script Test Data System Data 2 Test data selection System under Test TDC Test Data Container (TDC) 2013 SAP AG. All rights reserved. 45

46 Functional Regression Tests Test Reporting with SAP Solution Manager: Gap, Status and Progress Analysis Gap Analysis Quick Status Overview Test Status Reports Trend and Progress Reports Test Tool Logs Test Result Report Key benefits: Leveraging SAP Solution Manager for all status and results reporting, limiting investment in 3rd party tools 2013 SAP AG. All rights reserved. 46

47 Phase 1: Test Preparation 1c - Functional Regression Tests Test Ooption2 using SAP Solution Manager 7.1, HP Quality Center and SAP TAO

48 Functional Regression Tests Products and capabilities of Test Option 2 with SAP Solution Manager 7.1 Business Blueprint Change Impact Analysis Test Planning Test System Setup Test Execution Reporting Change Deployment Test Option 2 Solution Manager Business Blueprint Process documentation Business requirements Solution Manager BPCA Change Impact Analysis Test Scope Optimization Blueprint Adapter Defects / test results Quality Center Requirement Test Plan Business Process Tests SAP TAO Optimized SAP test scripts Accelerated maintenance SAP TDMS Setup of lean test systems Quality Center Test Lab Sprinter QTP Defect LoadRunner Load and performance testing 2013 SAP AG. All rights reserved. 48

49 Functional Regression Tests Manual Testing: HP Sprinter Floating windows with test step guidance for Tester Smart Defects Recording of User Actions (list) and Story Board (screenshots) Full integration with Quality Center for Test Script selection and results 2013 SAP AG. All rights reserved. 49

50 Functional Regression Tests Test Automation of Business Processes using SAP TAO Business Scenario Order to Cash Quotation Sales Order Delivery Goods Issue Billing SAP TAO Process Flow Analyzer SAP Quality Center by HP Business Analyst executes business transaction from SAP TAO Test Components for each SAPGUI subscreen with parameters for all fields upload Test Case* composed out of Test Components with parameters for all input fields File with Test Data for each input parameter (*) Note: Further test case refinement might be needed e.g. Test engineer may add more validation steps and consolidate the test case for performance improvement SAP AG. All rights reserved. 50

51 SAP TAO Test Execution with SAP TAO built test scripts SAP Quality Center by HP Test Script execution Test data file During test execution test data is entered into input fields of the SAP business process SAP Test System Test results Iteration Result # 1 Passed # 2 Passed # 3 Failed 2013 SAP AG. All rights reserved. 51

52 SAP TAO Test Automation of E2E Business Processes SAP TAO test scripts Test script variants can be built easily via additional Test Components, Flows and Test Data Scenario tests can be built easily through composition of the appropriate single test Variant 1 Quotation Sales Order V1 C1 C2 C3 Delivery Goods Issue Billing Test data Variant 2 Quotation Sales Order V2 C1 C2 C3 C4 Delivery Goods Issue Billing Test data SAP TAO Test Components 2013 SAP AG. All rights reserved. 52

53 SAP TAO Test Automation of E2E Business Processes Typical software changes affecting automated tests 1. Customizing changes for process step (optional field mandatory field,...) 2. Custom Code changes for process step (additional mandatory fields, additional screens,...) Variant 1 Quotation Sales Order V1 C1 C2 C3 Delivery Goods Issue Billing X Test data Test execution fails Variant 2 Quotation Sales Order V2 C1 C2 C3 C4 Delivery Goods Issue Billing X Test data SAP TAO Test Components Customer benefit: SAP TAO provides accelerated repair functionality through rapid and semiautomated exchange of central test components 2013 SAP AG. All rights reserved. 53

54 Functional Regression Tests Test execution, incident management and test results Goal Test Execution of manual and automated tests Test Lab execution and documentation of test results Defects managing incidents * additional license required 2013 SAP AG. All rights reserved. 54

55 Functional Regression Tests SAP Quality Center by HP Dashboards and Reporting Goal Reporting of test status and progress suitable for different types of users Scope Custom and role-based reports Coverage graphs with drilldown Benefits Comprehensive and holistic overview about test progress Out of the box reports Various reporting types usable for a wide range of users from manual testers to senior management * additional license required 2013 SAP AG. All rights reserved. 55

56 Phase 1: Test Preparation 1d - Performance Tests and Load Tests

57 Performance & Load Tests Definition of KPIs Definition of KPIs and target values Level 1 Performance & Load Test of key business processes End-to-end execution time including specific known queries Resource consumption of end-to-end scenarios including long running queries CPU consumption Memory consumption Degree of parallelization Throughput Volume restrictions Level 2 - Runtime behavior and resource consumption of HANA DB Statements Execution Time of specific known queries (SQLs / HTTPs) Resource Consumption of long running queries Measure CPU and memory consumption, and degree of parallelization 2013 SAP AG. All rights reserved. 57

58 Performance and Load Tests Definition of load profiles (1) Definition of Load Profiles Goal: Identification of Top 15 Processes causing ca. 80% of system load Approach 1 ST03N: Workload Monitor what s used in the system, elapsed time Download ST03N table to MS Excel Selection of Top Transactions causing ca. 80% of system load Double-Click: How many users STAD: Business Transaction Analysis what s happening in the system right now ST05: Performance Analysis SAT: Runtime Analysis Creation of SAP LoadRunner by HP scripts for these business processes 2013 SAP AG. All rights reserved. 58

59 Performance and Load Tests Definition of load profiles (2) Definition of Load Profiles Goal: Identification of Top 15 Processes causing ca. 80% of system load Approach 2 Interviews Frequently used business processes Business-critical business processes Custom developed business transactions Long Runners, but rarely executed (MRP Run, Fixed Asset Depreciation) How may users execute those processes in parallel (Average/Peak)? How often are they executed (Throughput: Sales Order / year, / month)? Creation of SAP LoadRunner by HP scripts for these business processes 2013 SAP AG. All rights reserved. 59

60 Load & Performance Tests E2E Test script example for CRM on HANA using SAP Loadrunner by HP Single Actions CRM on HANA E2E Script CRM on HANA Runtime Settings CRM on HANA Paramters CRM on HANA 2013 SAP AG. All rights reserved. 60

61 Load & Performance Tests Example results for CRM on HANA using SAP Loadrunner by HP Execution of the SAP LoadRunner by HP scripts in accordance to the load profiles defined: here: 1000 Vusers (= processes) here: step wise ramp-up of Vusers Think Times: 5 s Online resource consumption monitor during execution for defined monitoring point in SAP LoadRunner by HP here: Transaction Reponse Times Test Results: here: SQLs / hour here: HTTPs steps / hour Here: identified issue: Load Balancing 2013 SAP AG. All rights reserved. 61

63 Load & Performance Tests Runtime behavior & resource consumption of HANA DB SAP Recomendation: Drill Down Analysis on SAP HANA DB based on results of E2E Performance & LoadTests executed with SAP LoadRunner by HP Measuring the HANA Engines portion of the overall query response time Option 1: Expensive SQL statements Option 2: SQL trace Analyzing SQL execution of pure SQL statement in detail Analyzing SQL execution with the profile trace Capturing performance traces for Attribute, Analytic & Calculation Views Capturing a performance trace from the HANA Studio X-Server connection to the HANA database server and starting the HDB admin tool Capturing a performance trace with the HDB admin tool Finding and displaying a specific SQL query in the HDB admin tool Understand which engine executes which plan operations Understand the degree of parallel execution Drill down into single plan operations Analyzing the performance trace of an Analytic View Analyzing the performance trace of a Calculation View 2013 SAP AG. All rights reserved. 63

64 Phase 1: Test Preparation 1e - Service and Interface Tests

65 Service and Interface Tests Approach UI Layer / Custom Application A Custom Application B Application multiple use cases multiple use cases Layer Custom API Headless testing API test without application or UI DB Layer HANA Custom Table 1 Custom Table 2 SAP Table Test Tool: SAP Quality Center by HP Service Test Module Relevance for SAP HANA: Ability to test multiple use cases of custom-built interfaces with read / write access to custom and SAP defined tables residing in the SAP HANA database layer after technical migration 2013 SAP AG. All rights reserved. 65

67 Phase 2: Test Execution

68 Test Management for Suite on HANA migration projects Test execution during migration project Functional Tests Functional Tests by QM teams incidents corrections retest Q Gate IntegrationTests by Business Analysts Automated Regression Tests for E2E Business Processes Q Gate Functional Tests Adjustments and Optimization of Custom Code / Modifications for SAP HANA Performance & Load Tests Performance Tests for mission critical processes Comparison actual vs. baseline performance Optional: Load Tests and DB Stress Tests for large systems with very high throughput requirements Q Gate Performance Load Tests Service & Interface Tests Service and Interface Tests Functional tests of APIs / Services / Interfaces without UI Q Gate Interface Tests Reporting (test progress reports, status reports: open problems / resolution status) Sign-Off 2013 SAP AG. All rights reserved. 68

69 Test Activities after Go-live

70 Test Management for Suite on HANA migration projects Part 3: Regression Tests for HANA revision deployments Situation Error corrections for SAP HANA via SAP HANA Revisons Regression Tests Regression tests required at technical and / or process level Complete regression test not feasable due to time and effort constrains Approach Business Processes: Identify subset of functional tests for mission critical processes Technical level: Identify important business operations implemented via DB Views or Stored Procedures Define automated tests via functional test automation and service tests Execute automated tests as minimum regression tests to verify HANA revision 2013 SAP AG. All rights reserved. 70

71 Key Take Aways

72 Key Take Away System preparation Activate UPL as early as possible in your PRD systems Business Blueprint Use SAP Solution Manager capabilities, in case your blueprint is incomplete Functional test cases shall be assigned to business processes or process steps Test Scope 1 critical business processes Identify Business Process Experts Identify priority 1 business processes and assign custom attribute in your blueprint ABC classification of frequently used business transactions Test Scope 2 - Custom Code ABAP Test Cockpit (ATC) plus UPL for scope limitation SQL Monitor to identify important custom tables Functional Regression Tests Consider test automation Setup automated tests for all prio 1 business processes Prepare tests for CC adjustments Test Execution Assess how QM teams, test engineers and BPx can efficiently collaborate 2013 SAP AG. All rights reserved. 72

73 Test Management Overview and Detail Presentations and Offline Demos SAP Service Marketplace: Information Presentations Offline Demos More Information: -> Additional Information 2013 SAP AG. All rights reserved. 73

74 Best Practices / Next Practice: Regression Testing of SAP-centric Business Processes Leverage the experience of other SAP customers and find out what works best for them. Content Regresstion tests as integral part of the software change process 9 Recommendations for regression testing Customer use cases from Colgate-Palmolive, Ferrero, Sara Lee, ConAgra Foods, Soka-Bau Best Practice available at SDN: SMP: More Information Test Management 2013 SAP AG. All rights reserved. 74

75 Best Practices: Regression testing of SAP-centric Business Processes 9 Recommendations Business Blueprint Test System 1 Lean documentation of important business processes 2 Setup of lean test systems Creation of Regression Tests Change Impact Analysis 3 Test automation for critical business processes 4 Comprehensive test data for automated regression tests 5 Provisioning of easy to use manual tests 6 Change Impact Analysis: Risk-based test scope identification Test Execution 7 8 Unattended ( lights out ) execution of automated regression tests Manual test execution with guidance for business users Governance 9 Management of the change process 2013 SAP AG. All rights reserved. 75

76 201 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/vm, z/os, OS/390, zenterprise, PowerVM, Power Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, purescale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation. Linux is the registered trademark of Linus Torvalds in the United States and other countries. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered trademarks of Adobe Systems Incorporated in the United States and other countries. Oracle and Java are registered trademarks of Oracle and its affiliates. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems Inc. HTML, XML, XHTML, and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. Apple, App Store, ibooks, ipad, iphone, iphoto, ipod, itunes, Multi-Touch, Objective-C, Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc. IOS is a registered trademark of Cisco Systems Inc. RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry App World are trademarks or registered trademarks of Research in Motion Limited. Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik and Android are trademarks or registered trademarks of Google Inc. INTERMEC is a registered trademark of Intermec Technologies Corporation. Wi-Fi is a registered trademark of Wi-Fi Alliance. Bluetooth is a registered trademark of Bluetooth SIG Inc. Motorola is a registered trademark of Motorola Trademark Holdings LLC. Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. Sybase and Adaptive Server, ianywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase Inc. Sybase is an SAP company. Crossgate, m@gic EDDY, B2B 360, and B2B 360 Services are registered trademarks of Crossgate AG in Germany and other countries. Crossgate is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG. 201 SAP AG. All rights reserved.