Batch Schedule Performance Optimization

Transcription

1 Batch Schedule Performance Optimization Lydia Duijvestijn, Jan van Cappelle, Coen Burki March 2012 IBM Forum Amsterdam

2 Agenda Problem statement Solution outline The engagement The Source2VALUE portal Results and demo Next steps Q&A

3 From the news papers we infer that many IT problems can have their root cause in the batch... Internet banking unavailable Automatic collections doubly booked......and after that a faulty correction ATM s unavailable Pin payments in shops unavailable Stock exchange unavailable Payments between banks impossible

4 When we listen to the client, this is what we hear... Between 10:00 am and 4:00 pm we had bad online transaction response times. The time window to run my batch is too short. Processor utilization is too high. When we merge company A with company B, will there still be enough time to run our batch jobs? Our batch schedule is so complex that we don t know anymore how our applications work. We are worried that when there are problems we won t be able to resolve them quickly. Can you help us reduce our batch processing costs by 20%? There is no specific department responsible for our batch schedule, every department has it own standards and, no one thinks about the overall corporate perspective and requirements; how can we standardize? We want to clean up our batch schedule but do not know where to start. Can you help?

5 Why do companies have those problems? The core batch systems or Good Old Systems were built years or even longer ago New systems were built and connected to the old core Standard and guidelines were made, but never kept up to date and sometimes every department has its own standards. The world changed from 9 to 5 into 24*7. The question is where do we have a time slot to run the batch? Companies that used to have one mainframe now have one mainframe with hundreds of distributed platforms surrounding and connected to this mainframe And what can we do to help?

6 Agenda Problem statement Solution outline The engagement The Source2VALUE portal Results and demo Next steps Q&A

7 The solution is to implement Batch Governance and in the meantime start a Batch Optimization program Batch Governance To ensure alignment with the IT strategy and operational processes BSPO = Batch Schedule Performance Optimization Short term BSPO to answer questions such as What is the effect of a merger between two banks? What is the risk of an outage caused by batch window overrun Long term BSPO to cleanup the batch maintenance batchlog Agree and install standards and guidelines for batch processing Investigate batch critical timelines Prioritize issues and draw up roadmap Solve issues in order of priority Detect Long term BSPO is a cyclic approach, based on anti-pattern detection Test Analyse Change

8 Batch governance ensures alignment with enterprise architecture, IT strategy and operational processes IT Governance IT Management Enterprise Architecture Batch Governance Escalation reporting Align batch architecture with Enterprise architecture Priorities Batch Design Authority Operational Governance Problems Requests Technical control Quality assurance Advice Technical control Threatening trends Failure patterns RCA s Timelines Change the business Run the business Projects Design Authority Change Management Incident & Problem Management Application Support Operations

9 Batch Governance scope and tasks Scope: Design the end to end batch schedule of the enterprise. Document and enforce the standards and guidelines to run the batch schedule. Assure the quality of the batch schedule during design, build, operation, and change phases. Make architectural decisions about the batch schedule, especially for new features in advance. Make proposals for improvement. Tasks: Short Term - addresses the current batch problems and how they can be solved quickly.manages risk in the current environment and maintains the standards and guidelines up to date. Medium Term - creates and maintains the future architecture, including new machines and new applications. Involved with the creation of new solutions. Long Term - aligns the business strategy and the batch strategy and implements this strategy effectively. Develops strategic and innovative initiatives.

10 BSPO is based upon the detection of anti-patterns and replacement by patterns some definitions An anti-pattern (DON T) is similar to a pattern, but leads to negative consequences An anti-pattern describes what to avoid and how to fix a problem when you find it A pattern (DO) is a named solution to a recurring problem in a context A documented pattern specification communicates expertise from an expert to a novice, educating the novice about what makes the problem difficult and how to go about solving it It should provide the reader with enough information to decide whether this pattern is applicable to her problem, and enough understanding to be able to use it successfully A pattern describes how to solve a problem but does not advocate a particular implementation of the solution.

11 What makes the BSPO approach unique With BSPO IBM compares the schedule on one hand with the program sources (programming language, scripting language and data manipulation language) on the other hand. And checks on anti patterns. By using the Source2VALUE portal IBM is able to connect all kind of different data sources Batch schedule: TWS, CA7, Control-M Scripting language: JCL, BASH, Pearl Programming language: Cobol, PL1, C Database: IMS, DB2, Oracle Platform: Mainframe, Unix, I-series, Windows More then one schedule BSPO can cover the whole enterprise

12 When looking for performance (anti-)patterns we have to bear in mind the KPI s for batch performance Elapsed time Elapsed time is batch response time Elapsed time is the time taken from the start of a job to its end Elapsed time includes I/O time and all other types of wait The focus of the example BSPO engagement has been on elapsed time Throughput Throughput is defined as the number of units of work that a system can process in a given unit of time Capacity and resource utilisation Capacity is the amount of available system resources to process the batch Resource utilisation is the extent to which the capacity of a resource is used Resources are CPU, Memory, Disk, Network bandwidth

13 Focus of the example BSPO engagement has been on batch elapsed time Batch elapsed time is composed of three components Wait time Step 1: Eliminate wait times from critical path Work that should not be done Step 2: Eliminate unnecessary processing from critical path Work that needs to be done Step 3: Optimize necessary processing In case there are many critical timelines, the priorities have to be decided

14 Agenda Problem statement Solution outline The engagement The Source2VALUE portal Results and demo Next steps Q&A

15 Problem statement The merger of two banks will cause an increase in workload, potentially leading to blocking performance problems in multiple critical batch timelines Account balances not processed in time Stock exchange system unavailable Money transfer foreign currencies unavailable Bank-office applications unavailable Service to private clients unsatisfactory

16 Optimizing the batch is not exactly a one-dimensional problem Analyzing one timeline can be a nightmare, but companies tend to have several critical timelines... This client had over 600 critical timelines In the scheduling package there are over applications With over jobs running With more then half a million relations IBM has seen jobs with 75 steps

17 The client had good standards and did already several optimization projects in the past. 1 job has 1 procedure has 1 step Always top down re-runnable no problems with restart (temporary files) No time loss with restart Special resource only for 1 step When file is ready file can be delivered to next job 20% of components TWS, schedule JCL and procedure JCL and control cards are generated. Components (TWS, JCL and control cards) are build on development and promoted by tool to test, acceptance and production.

18 We proposed a four-phased short term BSPO approach Preparation Revisit user interface of the tool Create automated source collection mechanism Agree relevant measurement points Agree business critical endpoints Understand migration plans Collect, load, store & process Load data Collect and send sources Store and process sources Analysis & weekly intermediate reporting Perform high level analysis and store results Perform detailed analysis Report findings Define and capture new antipatterns Final reporting Compose end report and present results

19 Scope Analysis of critical timelines Four production timelines Core applications online point Analysis of dress rehearsal runs Several try-outs One full dress rehearsal run. Sources TWS Application DB Database TWS Current Plans TWS Track log information TWS Critical timelines Schedule JCL Procedure JCL SMF data Timeline analysis Critical timelines compared to baseline Anti-pattern analysis for critical timelines Whitespace analysis Analysis on TWS track log data Long running jobs Delayed endpoints Special request analysis Visits to the command-bridge

20 Wait times can be recognized as large gaps between jobs Anti-pattern Early start times [plan only]: If early start time incorrect remove Dependencies [+JCL+Program]: If dependency (technically) incorrect remove Waiting for special resources (VSAM datasets, DBs) [+JCL]: If incorrect use of special resources remove Waiting for normal resources (tape units) [+TWS DB]: If incorrect use of normal resources remove Job is waiting for unknown reason Threshhold has been exceeded (e.g. # of // servers) Deadlocks and abends No or little business involvement needed Source Gantt chart per critical path based on plan + actuals (track log) Cross reference per critical path based on PSB+JCL (for IMS) or Views+COBOL Sources+JCL (for DB2) Cross reference per gap within critical path based on planning+tws AD DB [optional]. Cross reference per gap within critical path based on planning + JCL + SMF 1. Cross reference between time, active job and workstation 2. Solution time analysis of abends, followed by zooming in on system log (777)

21 Work that should not be done takes on many forms Anti-pattern Duplicate copies of datasets and databases Remove from critical path or remove entirely Frequent backups Of datasets backups should be based on naming convention, e.g. CTY is backed up by system Of databases reduce frequency Stop and start of transactions Stop and start no more than once per night Stop and start of databases Keep 7x24 hrs online Source Lists, based on plan and JCL (parameterised service procedures) Lists, based on plan and JCL Lists, based on plan, actuals and JCL Lists, based on plan, actuals and JCL IEFBR14 jobs Remove IEFBR14 jobs [optional, maintenance work] Lists, based on plan, actuals and JCL Business involvement needed to judge (in)correctness of operations

22 Work that must be done must be done as efficient as possible (Anti-)pattern Find the 20% jobs that constitute 80% of the work Source Analysis of the actuals, joblogs Job optimisation Remove unnecessary resource allocations Remove wait times Remove redundancy and duplications Remove deadlocks Joblogs, sources, Input / Output datasets, SMF Parallellisation where possible Run sequential jobstreams in parallel instead Split jobs into multiple parallel jobs Sources, Input / Output datasets Heavy business / application maintenance vendor involvement needed

23 16 Antipatterns were analysed and the list is growing ID Anti_SW_000 Anti_SW_001 Anti_SW_002 Anti_SW_003 Anti_SW_004 Anti_SW_005 Anti_SW_006 Anti_SW_007 Anti_SW_008 Anti_SW_009 Anti_SW_010 Anti_SW_011 Anti_SW_012 Anti_SW_013 Anti_SW_014 Anti_SW_015 Name Superfluous relation Superfluous special resource Superfluous early start time Redundant copies of files Superfluous dummy steps Redundant image copies (backups) Unused code (COBOL) Unused code (JCL) Redundant start and stop (transactions) Redundant start and stop (databases) Redundant change status (tablespaces) Offline programs (DBB/DLI) GSAM init. In critical path Immediate start when after extend current plan Wait for input BMP Runcycle not in line with jobname

24 Agenda Problem statement Solution outline The engagement The Source2VALUE portal Results and demo Next steps Q&A

25 Source2VALUE Portal detects and shows antipatterns across platforms (TWS, JCL, Cobol,..) Anti-patterns are translated to signals on the dashboard Whitespace shows gaps in critical timeline

26 Source2VALUE Portal increases analyzability and saves time (and money) Different views: Dashboard view Timeline view (job flow visualization) Cross Reference view Source view Click All elements are hyperlinked across platforms Click Two starting points for analysis The dashboard for analysing anti-patterns The job flow visualisation for white space analysis

27 Source2VALUE Dashboard shows all jobs with configured antipatterns Optimization opportunity: 7,5% of the jobs in this critical timeline contain image copy jobs

28 Source2VALUE Timeline shows gaps in the plan Timeline Contains slots of continuous work in a critical timeline. Whitespace means an opportunity to optimize Analyze white space Timeslot Gantt Contains individual jobs and dependencies within a timeslot Used to find the starting jobs in a timeslot to start further analysis Used to detect suspected dependencies and durations Click

29 Source2VALUE lets you focus on optimization opportunities Adaptable New anti-patterns can be added New sources can be added (e.g. PSB) Configurable The dashboard can be configured depending on the desired focus Repeatable Trend analyses Difference analysis <expression> <or> <gt metric="delta_target" value="1"/> </or> <or> <gt metric="delta_deadline" value="1"/> </or> <or> <gt metric="critandadstarttime" value="1"/> </or> <or> <gt metric="no_ims_ic" value="1"/> </or> <or> <gt metric="no_db2_ic" value="1"/> </or> <or> <gt metric="no_ims_ss" value="1"/> </or> <or> <gt metric="no_db2_ss" value="1"/> </or> <or> <gt metric="no_dummyjobs" value="1"/> </or> </expression> Dashboard configuration in XML-file

30 Agenda Problem statement Solution outline The engagement The Source2VALUE portal Results and demo Next steps Q&A

31 Example: Dummy jobs in Client database Sort jobs on anti-pattern and start analysis 25 dummyjobs in critical timeline Click for job details List of jobs sorted on anti-pattern

32 Example: Dummy jobs in Client database Analyze source code Dummy job can be removed and dependencies adjusted

33 Example: 17% image copies, start/stops and dummy jobs in critical timeline Sort jobs on anti-pattern and start analysis 17% of jobs in critical timeline is not core functionality ( )/ sequential IMS Image copies in one job

34 Example: Gap in timeline due to early start in Triple A timeline Analyze gap by investigating timeslot after gap Click on timeslot after gap to view details

35 Example: Gap in timeline due to early start in critical timeline Investigate job details to determine early start FTP incoming stock exchange Job has early start defined in Application Database. Business has to be consulted to determine optimization possibilities

36 Extra: Standalone job with no early start is potential risk Investigate why job starts at 3:30 Job pdofi60t is a GSAM initialization job that runs directly after plan extend at 3:30 instead of at 12:00... Potential risk when previous plan is not finished yet.

37 ET online point : 12-4 to 16-4 and 19-4 to 23-4 (bottom) Baseline 16 january to 11 februari Trimmed averages Baseline startdate Baseline charactistics 16-jan (See Baseline enddate 11-feb Critical timeline compared to baseline e.g. ET online point for DR4-C1 and DR4-C2 dates Total nr of jobs available Total nr of jobs analysed Total nr of jobs filtered out Total nr of unique jobs analysed

38 Agenda Problem statement Solution outline The engagement The Source2VALUE portal Results and demo Next steps Q&A

39 Notices This information was developed for products and services offered in the U.S.A. Note to U.S. Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces IBM Corporation

40 Trademarks This presentation contains trade-marked IBM products and technologies. Refer to the following Web site: IBM Corporation

41 Participate in the System z Expert and Superhero contest! Fill in your answer to the question below on the scorecard and deposit your card in the box! An example of a performance anti-pattern impacting batch elapsed time is a.the presence of dummy jobs in the schedule b.redundant use of image copies c.existence of obsolete relations in the schedule d.all of the above IBM Corporation

42 More information on zenterprise IBM zenterprise landing page: IBM zenterprise 114 (z114): IBM zenterprise Events Landing Page: IBM software for zenterprise: IBM System Storage: IBM Global Financing: IBM Services for zenterprise: IBM zenterprise / System z Redbooks Portal: IBM Corporation

43 Traditional Chinese Thai Russian Thank You English Bedankt Nederlands Merci French Obrigado Gracias! Spanish Arabic Brazilian Portuguese Danke German Simplified Chinese Japanese IBM Corporation