7 deadly sins in WLC

7 deadly sins in WLC Fabio Massimo Ottaviani Demand Technology Software Abstract In mainframe environments the software products prices have been historically based on the computing capacity of the central processor complex (CPC) on which the software was running, a pricing policy valid both for IBM and independent software vendors. This pricing policy, widely accepted in the market, has become less and less valid over time, mostly because the business advantages related to an increase of capacity don t match the increase in software related costs. Moreover, the cost of mainframe software compared with that of the software of other platforms has become the major deterrent to mainframe investments. Another factor has been the explosive growth of new workloads, particularly in the area of e- business applications; this has probably been the key element that forced IBM to propose a different policy to make the mainframe environment more affordable for the deployment of these new workloads. This new policy is called Workload License Charges (WLC) and can be: Fixed, named Fixed Workload License Charges (FWLC) Variable, named Variable Workload License Charges (VWLC). The VWLC policy is the real innovation and allows paying software licenses based on the CPU usage (the maximum monthly value of the 4-hour rolling average) instead of the CPC capacity. It s important to be aware that a cultural change in the way systems are managed is required, in order to take full advantage of a WLC contract. Otherwise some of the current behaviors will become deadly sins not allowing to get the cost savings expected from company management. In the first part of this presentation, the basic WLC concepts, the metrics and techniques needed to calculate the 4-hour rolling average will be discussed. In the second part the 7 deadly sins in WLC will be presented.

INDEX 1 INTRODUCTION 3 2 WHAT CPU USAGE DOES REALLY MEAN IN VWLC 4 3 HOW THE SUB-CAPACITY REPORTING TOOL WORKS 5 4 THE DEADLY SINS 6 4.1 Sin 1 Products Topology 6 4.2 Sin 2 CPU overhead 6 4.3 Sin 3 Looping tasks 7 4.4 Sin 4 Low priority CPU intensive tasks 8 4.5 Sin 5 System Tests 9 4.6 Sin 6 Application Stress Tests 10 4.7 Sin 7 Artificial peaks 10 5 CONCLUSIONS 12 6 BIBLIOGRAPHY 13 APPENDIX A 14 APPENDIX B 15 2

1 Introduction The VWLC policy is the real innovation of the new software pricing policy proposed by IBM in the mainframe environments. It s applicable to the major software product as CICS, DB2, IMS, MQ Series and Websphere 1 when they run on z/os on a zseries machine. VWLC allows you to pay software licenses based on the CPU usage instead of the CPC capacity. It s important to understand what CPU usage does really mean. The CPU usage is calculated based on the MSU (million CPU service units) measurement unit. Every 5 minutes the Workload Manager (WLM) inside each LPAR calculates a 4-hour rolling MSU average and writes that to an SMF 70 record. See Chapter 2. The Sub-Capacity Reporting Tool (SCRT) at the beginning of each month takes in input the SMF records of the previous month and provides the LPAR utilization capacity. The LPAR utilization capacity is the highest sum of the measured 4-hour rolling MSU averages for the LPARs in the CPC. See Chapter 3. It s important to be aware that a cultural change in the way systems are managed is required, in order to take full advantage of a WLC contract. Otherwise some of the current behaviors will become deadly sins not allowing to get the cost savings expected from company management. See Chapter 4 1 For Websphere products the applicable option is IPLA; this option allows for sub-capacity pricing and for the scope of this document can be considered in the same way than VWLC. 3

2 What CPU usage does really mean in VWLC The CPU usage is measured in MSUs. The MSU value represents millions of CPU (TCB+SRB) service units per hour and it has to be calculated from the raw (un-weighted) service units. An interesting thing to note is that every CPC has a commercial MSU power established by IBM; this MSU value is consistently different from the one normally used for performance and capacity planning activities based on the R723MADJ field of the SMF72 records. The commercial MSU value is the one used to calculate software costs and it s based on the value of the SMF70CPA field in the SMF 70 records. SMF70CPA allows calculating the number of CPU service units per second for each processor of the CPC. This value is constant and is not based on the number of logical processors assigned to each LPAR. Every 5 minutes the WLM inside each LPAR, calculates a 4-hour rolling MSU average of the MSU used and writes it into the SMF70LAC field of the SMF 70 record (at the user defined RMF interval). The reason why the WLM uses a 5 minutes interval is mainly related to the defined capacity issue and it s beyond the scope of this paper. The values recorded in the SMF70 records determine the LPAR MSU utilization and are needed to calculate the bill for z/os. To calculate the bill for all the other products it s important to know when each product was active inside the different LPARs. Each product bill will not be based on the product CPU utilization in the LPARS but on the sum of the utilization of all the LPARs where the products were running. The information concerning products activity is recorded inside the SMF89 records 2. 2 SMF 89 records are not produced for all the products; the SCRT manual explains how to manage this issue. 4

3 How the Sub-Capacity Reporting Tool works The Sub-Capacity Reporting Tool (SCRT) is a free IBM tool that reports capacity utilization for sub-capacity eligible products that run on z/os. The sub capacity eligible IBM products are reported in Appendix A and B. The SCRT at the beginning of each month takes input from the SMF 70 and 89 records 3 and determines the used capacity by examining, for each hour in the reporting period: The rolling 4-hour average utilization, by LPAR Which eligible products were active in each LPAR SCRT then cross-references LPAR utilization and product execution by LPAR to determine the maximum concurrent LPAR rolling four-hour average utilization - the highest combined utilization of LPARs where each product executes during the reporting period. 4 The following table shows an example of how the SCRT calculates the LPAR utilization capacity for different products and LPARs combinations. 4-hour rolling average z/os CICS IMS HOUR LPAR1 LPAR2 LPAR1 LPAR2 SUM LPAR1 LPAR2 SUM LPAR1 LPAR2 SUM 1 100 333 100 333 433 0 333 333 0 333 333 2 120 321 120 321 441 0 321 321 0 321 321 3 112 345 112 345 457 0 345 345 0 345 345 4 118 367 118 402 520 0 402 402 0 402 402............ 718 127 348 127 348 475 127 348 475 0 0 0 719 133 299 133 299 432 133 299 432 0 0 0 720 122 300 122 300 422 122 300 422 0 0 0 MAX 520 475 402 Figure 1 The MAX value is the highest utilization determined from the sum of the utilization for all LPARs in which a particular product ran in a given hour. It is not the sum of highest utilization for individual LPARs in which a particular product ran during different hours. In this example, the peak value of the month for z/os is in hour 4. It is the sum of the z/os utilizations across all the LPARs during hour 4, or 520 MSUs. The peak value for CICS is in hour 718 5. IMS ran only in the first part of the month in LPAR2 so its utilization value is 402 MSU. There are specific rules the SCRT uses to automatically determine the reporting period and calculate the percentage of data collected versus the expected data of the period. You have to be aware of these rules because if that percentage is less than 95%, you have to justify it. 3 To comply with WLC terms and conditions, the input data sets must contain one reporting period of SMF 70 and 89 records for all the z/os images on a zseries CPC. 4 From the Using the Sub Capacity Reporting Tool manual. 5 In the example a month of 30 days with 24 hours data collected each day is considered. 5

4 The deadly sins This chapter points out some behaviors you have to avoid in order to get the cost savings allowed by VWLC. All the situations (sins) described here are only examples but similar situations have been verified in many sites during the last year. 4.1 Sin 1 Products Topology It s very important to be aware of the possibility to pay less or more depending on the LPAR where products run. In the past whatever LPAR in a CPC you ran a product the price to pay was always the same and was only determined by the CPC power. In VWLC you have the advantage to pay only for the power used (in the 4-hour rolling average) by the LPARs where products run but you have to be careful in order to take advantage of this opportunity. IBM 2084-306 (395 MSU) LP1 270 MSU IMS DB2 LP2 90 MSU WAS DB2 Figure 2 In the CPC in Figure 2 there are 2 LPARs active. LP1 uses 270 MSU, most of them for IMS and DB2 applications. In LP2 the major workloads are WAS and DB2. A new application has been acquired based on CICS so a CICS subsystem has to be started in one of the two partitions. Before VWLC, starting CICS in one partition or in the other would not have any impact on CICS software cost. You had to pay for the entire CPC capacity. In VWLC starting CICS in LP1 will be much more expensive because CICS will be paid for its consumptions plus the 270 MSUs already used by IMS and DB2. If started in LP2 the cost of CICS will be based on its consumptions plus the 90 MSUs already used by WAS and DB2 6. 4.2 Sin 2 CPU overhead There are so many papers and flashes available in CMG proceedings, in IBM manuals and in the Internet describing how to reduce the CPU overhead, that it is really surprising how many overhead problems can still be found at many customer sites. Before VWLC the overhead was essentially a performance issue and it was addressed only in a severe lack of capacity situation. In VWLC all the MSUs wasted will be paid at the end of the month so the CPU overhead has to be reduced and controlled as much as possible. 6 An equal cost for IMS and WAS licenses is supposed in this example. 6

In the following table the more frequent overhead reasons are reported. PRSM overhead DB2 hiper pools overhead Too many LPARs, too many logical processor; all the processors are often assigned to each LPAR also if not needed Many customers are still using DB2 hiper pools; the conversion of the DB2 hiper pools into Data Spaces can allow to save until 30% of the CPU seconds used by the DBM1 address spaces RMFGAT overhead CACHE parameter in RMFGAT; when thousands of device are active the overhead can be until 15 MIPS for each partition; most of the customers confused this parameter with the RMF Monitor I CACHE parameter needed to produce the SMF 74 subtype 5 records Monitors overhead WLM overhead Sampling too aggressive; normally it s better to change the cycle default values only temporary for specific analysis Excessive MAXTASK values in CICS regions managed at transaction level Figure 3 4.3 Sin 3 Looping tasks Dozen of monitors are available at every customer site to control the system and all the major subsystems. Furthermore hundreds of automation solutions have been developed since many years to immediately react to correct and eliminate any kind of anomaly. So it s strange to see how many times different kind of tasks enter a loop and stay in the system for hours, sometimes for days, before being intercepted and cancelled. The basic thresholds to control looping tasks are: Percentage of CPU usage for TSO, STC and BATCH address spaces; Percentage of CPU usage summed across all the occurrences of the same address space name for OMVS workloads; Percentage of CPU usage summed across all the occurrences of the same address space name for STC workloads; Amount of CPU seconds used for CICS, DDF and IMS transactions. The detection of a looping task is normally very quick if a system is heavily loaded; in this situation the loop causes visible performance degradation and the consequent reaction of instruments and people. If the system is lightly loaded the reaction can be very slow and the task can loop for hours before being cancelled. 7

It s important to understand that, in VWLC, a slow reaction can have a direct and heavy impact on the monthly software bill. The SCRT manuals says: Customers have the primary responsibility for preventing uncontrolled loops, operator errors, or unwanted utilization spikes. However, IBM understands that occasionally situations that could not be prevented (especially situations related to disaster recovery) may cause exceptional utilization values. In these situations, IBM does not expect our customers to pay for the increased utilization associated with the unusual situation. Please use your best judgement to determine if an unusual situation has occurred. IBM does not publish a list of unusual situations because they will by their nature be unpredictable. IBM trusts your judgement when you indicate an unusual situation has occurred. However, if you repeatedly override the Tool MSUs (for example, you always have unusual situations occur at quarter close) or if large discrepancies are found during IBM audits (IBM Call Home TSAD data is compared with data from subcapacity reports), IBM will contact you at the time the pattern is noticed. So it s important to react as fast as possible to loop situations but it s also important to track and log all the loop events that happen because this information can be eventually useful to correct the MSU values at the end of the month. 4.4 Sin 4 Low priority CPU intensive tasks In every system there are many tasks (essentially batch jobs) that are CPU intensive and normally run at low priority using only the CPU not used by the more important workloads. When the system is lightly loaded, these tasks can however consume huge amount of resources rising the 4-hour rolling average. It s not infrequent to see systems where the higher values of the 4-hour rolling average are measured when the more important workloads are not active. It s very important to give to these kind of tasks only the resources needed to meet their service level objectives and to avoid increasing software costs without no substantial benefit. An important group of this kind of tasks is the one related to the SMF processing. In Figure 4 the contribution to the 4-hour rolling average of a very efficient SMF processing at a customer site is reported. The global 4-hour rolling average (not reported in the figure) was about 2.700 MIPS while the SMF part reaches 80 MIPS (about 3% of the total). SMF processing CPU consumptions 4-hour rolling average 90 80 70 60 50 40 30 20 10 0 16 14 12 10 8 6 4 2 0 Mips Figure 4 MSU 8

The SMF processing related applications are good candidates for downsizing. There are many customers in the world processing SMF (for Cost Accounting, Service Levels, Performance and Capacity Planning activities) on Windows and Unix platforms. The major benefits are: Software licence costs savings; CPU consumptions reduction; Dedicated environment. In general, one possible option to control the amount of resources used by low priority tasks is to implement WLM resource groups. Automation execs can be written to control the 4-hour rolling average value and move the CPU intensive low priority tasks in service classes associated to different resource groups when needed. 4.5 Sin 5 System Tests Some times the CPU usage peaks are related to system programmers test activities of a new system, subsystem or product release. In Figure 5 the CPU usage profile, expressed in MIPS, of the CPC (IBM 2084-310) is reported. ZOSD, ZOSC and ZOSF are production partitions, LINX hosts the z/linux environment and TEST is dedicated to system programmer s tests. After the IPL at 1 am, the TEST LPAR started to use about one processor until the production partitions, that have higher PRSM weights, compressed it in the peak hours from 10 to 12 am. CPU usage profile LPAR 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 ZOSD 783 659 935 721 678 782 612 706 795 1300 1763 1916 1995 1494 1517 1751 1739 1499 1094 1006 2041 1889 1049 663 ZOSC 494 343 426 605 568 818 807 520 703 1122 1333 1272 1010 810 793 889 909 1183 644 559 1087 1243 918 568 ZOSF 183 179 157 107 96 67 112 106 118 159 262 260 381 520 283 279 244 258 111 122 126 120 117 68 TEST 209 5 277 357 357 354 354 358 354 307 115 80 120 268 313 242 257 256 335 333 155 142 341 359 LINX 32 29 29 29 29 29 29 37 38 35 37 35 33 30 30 43 30 30 29 28 29 29 28 28 Figure 5 The system programmers looking at this report understood something was wrong and detected a loop in the new version of a batch-scheduling product. They cancelled the product and opened an incident to the software vendor. In Figure 6 it s possible to appreciate the effect of this loop in the 4-hour rolling average values for the CPC. MSU usage (4-hour rolling average) LPAR 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 ZOSD 158 113 122 131 131 126 103 99 120 153 188 208 226 213 205 211 202 206 195 183 211 236 248 230 ZOSC 121 87 69 68 66 81 95 93 105 111 110 116 111 99 96 100 95 99 91 82 95 112 105 95 ZOSE 18 19 21 22 19 16 13 14 15 21 34 47 57 58 51 44 37 35 29 23 20 19 20 18 TEST 25 20 15 32 37 50 53 53 53 51 42 32 23 22 29 35 40 40 41 44 40 36 36 37 LINX 4 5 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 Total 326 245 233 260 261 283 274 271 306 350 390 419 434 410 400 410 395 402 379 355 391 428 435 407 Figure 6 The highest value is 435 MSUs and it s measured at 10 pm. The TEST LPAR contribution is 36 MSUs. 9

4.6 Sin 6 Application Stress Tests An impact on the 4-hour rolling average, and consequently on software costs, can come from application tests before production. In the case of application stress tests this impact can be severe. It s not possible to avoid this kind of activity but accurate planning and scheduling can probably avoid a raise in the monthly software bill Application tests have to be planned in the right partition, at the right time of the day. In the CPC in Figure 6 the hours before 9 am could be the right time to run a test; the right place could be the TEST partition. It s advisable to cap the TEST LPAR to limit the amount of power that can be used. If the application test needs more power (i.e. 200 MSUs), it has probably to be scheduled on Saturday or on a day where the overall load is very light. 4.7 Sin 7 Artificial peaks In many data centers there are CPU usage peaks not related to real business needs but only to organizational and historical reasons. A frequent issue is related to batch scheduling; there is normally an open the gates approach that very often can cause the highest values in the 4-hour rolling average also if there is not a specific service objective need. The more dangerous sin of this type is, however, when the peak has been caused by long system unavailability. The graph in Figure 7 reports a simulation showing the impact of 2 hours of system unavailability (from 8 to 10 am). The artificial peak generated by the system crash has caused the concentration of all the waiting workloads in the hours following the system restart (see the Abnormal bar). The 4- hour rolling average is much higher than it would be in the Normal situation. MSU used 4 hour rolling average MSU 600 500 400 300 200 100 0 11 12 13 14 15 hour Figure 7 Normal Abnormal 10

It s obvious that a system unavailability of this type, if the system hosts business critical applications, may have a much higher impact on the company business than the one related to the software costs. However it is a paradox to pay a higher bill in a month when such a situation occurs and it could be a hard thing to explain to your executive manager. What can be done is to prepare an emergency procedure to slow all the LPARs and non business critical workloads, to reduce the impact of the artificial peak following a system restart. 11

5 Conclusions A cultural change in the way systems are managed is required in order to really take advantage of the costs savings promised by WLC. Many of the current behaviors have to be changed to avoid the sins described in this paper and many others not mentioned. Beyond the economical advantages, the more interesting aspect of WLC is that a new professional figure will probably emerge to take the burden of software capacity planning and act as a bridge between the performance analysts and the budgeting department. 12

6 Bibliography Books SA22-7506-06 Planning for Workload License Charges SG24-6522-09 Using the Sub Capacity Reporting Tool - SCRT Version 7.1 SA22-7630-01 MVS System Management Facilities (SMF) SC33-7991-01 Resource Measurement Facility Report Analysis Papers Offloading SMF data processing from the mainframe Mark Cohen Austrowiek, Fabio Massimo Ottaviani CMG-Italia 2003 proceedings IBM's WLC impact on Performance and Capacity Planning Al Sherkow CMG U.S.A. 2003 Proceedings 13

Appendix A Sub-Capacity Eligible MLC Products Product ID Product Name SMF89? 5694-A01 z/os Version 1 Yes 5655-G52 z/os.e Version 1 (EWLC only) Yes 5655-147 CICS TS for OS/390 Yes 5697-E93 CICS TS for z/os V2 Yes 5655-018 CICS/ESA V4 Yes 5695-DB2 DB2 FOR MVS/ESA V4 Yes 5655-DB2 DB2 for OS/390 V5 Yes 5645-DB2 DB2 UDB for OS/390 V6 Yes 5675-DB2 DB2 UDB for OS/390 V7 Yes 5625-DB2 DB2 UDB for z/os V8 Yes 5695-176 IMS/ESA Version 5 Yes 5655-158 IMS/ESA Version 6 Yes 5655-B01 IMS V7 Yes 5655-C56 IMS V8 Yes 5655-J38 IMS V9 Yes 5695-137 MQSeries MVS/ESA Yes 5655-A95 MQSeries for OS/390 V2 Yes 5655-F10 MQSeries for OS/390 V5 Yes 5648-A25 COBOL for OS/390 & VM V2 No 5655-G53 Enterprise COBOL for z/os and OS/390 No 5655-H31 Enterprise PL/I for z/os and OS/390 No 5697-ENV IBM Tivoli NetView for z/os No 5655-K36 Lotus Domino for z/os V6 Yes 5655-B86 Lotus Domino for S/390 V5 No 5645-006 System Automation OS/390 V2 No 5645-005 System Automation for OS/390 No 5697-WSZ Tivoli Workload Scheduler for z/os No 5655-043 Tivoli NetView PM No 5697-B82 Tivoli NetView for OS/390 No 5697-OPC Tivoli OPC No 5655-B22 VisualAge PL/I OS/390 V2 No 5706-254 Query Management Facility V3 No 5655-H32 Debug Tool for z/os and OS/390 No 5655-L24 Debug Tool for z/os V4 No 5695-068 Airline Control System (ALCS) V2 Yes 14

Appendix B Sub-Capacity Eligible IPLA Products Product ID Product Name SMF89? 5655-I35 WebSphere App Server for z/os V5 Yes 5655-F31 WebSphere App Server for z/os and OS/390 V4 Yes 5655-K12 WebSphere Portal for z/os and OS/390 V4 No 5655-I03 WebSphere Host Publisher V4 for zseries No 5655-I58 WebSphere MQ Integrator Broker for z/os V2 No 5655-G97 WebSphere MQ Integrator for z/os V2 No 5655-BPM WebSphere MQ Workflow for z/os V3 Yes 5655-I40 WebSphere Data Interchange for z/os V3 Plan 5655-K60 WebSphere BI Message Broker for z/os V5 No 5697-I11 WebSphere BI Message Broker with R&F for z/os V5 No 5655-K57 WebSphere BI Event Broker for z/os V5 No 5655-J67 WebSphere Studio Application Monitor V1 No 5655-I49 WebSphere Studio Asset Analyzer V2 No 5655-I14 WebSphere Studio Workload Simulator V1 No 5655-L21 WebSphere Studio Asset Analyzer V3 No 15