Best Practices for DB2 on z/os Schema Management. By Klaas Brant and BMC Software

Transcription

1 Best Practices for DB2 on z/os Schema Management By Klaas Brant and BMC Software

2 Page 2

3 CONTENTS Chapter 1: Introduction to change management... 9 Why do we need change?... 9 Change drivers... 9 What can we change? Infrastructure/software upgrades Migrating to a new release of DB2 consider DB Applying DB2 maintenance Availability and performance improvements Access paths How BMC can help Indexes Virtual indexes Clustering indexes How BMC can help Tablespace How BMC can help Locking Utilities Business application enhancements and fixes Application errors (bug fixing) Data pollution and data corruption How BMC can help New applications and retired applications New or changed business logic Page 3

4 Compliance The impact and risk of making changes Incorrect implementation Incorrect backout Why is it so hard? Compliance with corporate standards Change management Governance and audit Ability to reproduce old data Chapter 2: Managing application changes in DB Lifecycle management The First Law System upgrades a conceptual best practice Independent from application Versioning and fallback Release policy Implement and backout Phased vs. big bang How BMC can help Test your changes How BMC can help Problems for system changes Chapter 3: The complexity of DB2 structure changes The three type of changes Type 1: Simple change minimal to medium impact Type 2: Online change - initiates DB2 object versioning Page 4

5 Type 3: Complex change Unload, Drop, Create and Load (UDCL) How BMC can help Unavailability during the deployment How BMC can help Keep your data healthy How BMC can help Risk of altering scripts How BMC can help Deployment and impact of changes A special undo scenario Is an audit needed? What s new in DB How BMC can help Chapter 4: Managing DB2 security Object qualifier CATMAINT for updating schemas System administrator Trusted context and roles How BMC can help Chapter 5: Managing the DB2 catalog Catalog REORG Verify catalog consistency DBDs and SYSCOPY SYSIBM.SYSUTILX Invalid or inoperative packages Package versions Page 5

6 DB2 plan stability View regeneration errors How BMC can help Catalog queries Chapter 6: Schema management What can we change and how does it affect us? Data sharing Anatomy of a change Application changes: a closer look Program only changes, no data structure changes DB2 data structure changes with minimal impact DB2 data structure changes with impact: online change The high impact change The impact of change The risk of implementing a change DB2 10 improvements Conclusion Chapter 7: Simplify DB2 for z/os schema management Change management - a necessary evil BMC CHANGE MANAGER for DB Manage changes Migrate data structures Migrate data structure changes only Recover data structures Record and control changes Feed back changes Page 6

7 BMC CHANGE MANAGER for DB2 components Baseline Compare CM/PILOT CDL and DDL files The power of parallel performance Exploiting hardware and software capabilities Summary Page 7

8 Page 8

9 Chapter 1: Introduction to change management Most businesses consider data to be the lifeblood of their organization. Its availability and integrity are considered an imperative to making decisions and providing service to internal and external customers. The 21st century has brought dynamic and wide scale changes to how business is conducted. With data having achieved asset status, the agility of managing one s data and responding to change at the same time is a contributing factor to profitability and growth. Why do we need change? Before answering that question, let s answer another what is change? The American Heritage Dictionary defines change in its noun form as: The act, process, or result of altering or modifying and the replacing of one thing for another. Every business strives to succeed in an environment of constant change. Success can be measured by how well a business handles change - in its management, in its market, in its products, and as a result in its business applications. Which brings us to the purpose of this book: to help DB2 for z/os database administrators plan and implement change with a minimum of risk and a maximum of success. That success can be measured by managing change in a manner that maintains Data integrity and object synchronicity With a minimum of downtime Within the requested time period Change drivers An organization that has chosen DB2 for z/os as its database management system has made a prudent choice when it comes to ensuring data integrity and availability. Supporting DB2 data structures and objects (tables, indexes, data sets, and so on) often requires change. The tasks required to implement changes and the magnitude of their impact can vary from minimal to very complex, carrying with them the burden of application downtime, risk of data loss (in case of errors) and loss of customer satisfaction. Changes are driven by the business and sometimes technology. Here are some examples: Page 9

10 Business driven o o o o Changes to product requirements or business rules Mergers and acquisitions Regulatory compliance Errors in business application logic Technology driven o o o System upgrade Discontinued features of DB2 DB2 features (such as data sharing) Most changes to production applications are planned outages, except emergency fixes. Most organizations have implemented some form of change management process that includes procedures, documentation, and approvals. A good process includes well-defined implementation and backout plans. A change becomes an unplanned outage when things go wrong during the implementation. Therefore, a change management process needs an escalation plan as well. Unplanned outages occur because of hardware failures and data corruption, the latter sometimes as a result of an application or data related change (such as logic error in program). They can also be caused by an error when making a change to a DB2 object. To address and implement a change, a DBA may need to take any number of actions, such as: adding views, dropping an index, changing a data type, changing buffer pools, adding new columns, and so on. While some changes can be made in an ad hoc manner, maintaining data integrity and availability requires a solid change management process. Not investing in a solid change process is asking for trouble. Only with the right procedures and tools can you mitigate the risks of change. What can we change? One thing is certain: you will always have things to change in DB2 for z/os. Change is influenced by logical and physical structures, operating systems and physical states, DB2 object changes, and SQL statement constructs, as well as business application coding and process. Let s look at the main areas for change in DB2: Infrastructure/software upgrades Page 10

11 Availability/performance improvements Business application enhancements/fixes Infrastructure/software upgrades Whenever we introduce new hardware or software to the system infrastructure, we introduce change. Many of these changes, like a new controller or re-routing a network, usually go unnoticed or have minimal impact on DB2 and limited impact on your business applications. However, the introduction of a DB2 data sharing environment or a new release of DB2 for z/os can have a huge impact and trigger a significant amount of change. Migrating to a new release of DB2 consider DB2 9 The implications of change when migrating to a new release of DB2 can be daunting. You must first understand all stages of the migration and each stage s impact on the DB2 infrastructure and the business application. Each stage has actions that impose limitations for exploiting new functions and limitations for fall-back. Catalog changes play a significant part in any release migration. As with previous releases of DB2, additional columns are always added to existing catalog tables, columns are deleted, and new catalog tablespaces and tables are created. The chart below depicts the changes over past releases. Note the number of changes for DB2 9 - and that does not include the XML schema repository that resides outside the DB2 catalog! DB2 Version Tablespaces Tables Indexes Columns Table Check Constraint V N/A V N/A V V V V V V10 extreme restructuring Page 11

12 Many DBAs have established queries, procedures, and home-grown tools that rely or access on the DB2 catalog. All of these must be reviewed with each new release for accuracy and changed if needed. Even if you only use vendor tools, you must install a new version of the tool to support the new release. With every migration, some facilities and functions go away. For example, DB2 9 removed DB2- managed stored procedures and the ability to create simple tablespaces. IBM provides a list of withdrawn features, so you have plenty of time to change your structures and procedures. DB2 9 introduced over 80 feature enhancements in the areas of availability, stored procedures, SQL enhancements, security, application enhancements, performance and scalability, utilities, XML and data sharing, including: RENAME COLUMN Universal table spaces Native SQL stored procedures New data types New built- in functions New security roles Optimistic concurrency control MERGE SQL statement Operational changes in 15 utilities Implementing even one of these features introduces change to DB2 objects and/or application programs. Realistically, very few organizations could envision adopting all of the new features in a release of DB2, but almost every organization will implement at least some. Initially, you can limit or prohibit the use of new functionality until the environment is stable. Ultimately, there will be an impact to at least one of your business applications because business owners want to exploit new features to deliver better service to customers. During the migration period your development and test environments are temporarily out of sync; perhaps your test system is already on DB2 9 New Function Mode (ready with migration) and the production system is still in Conversion Mode (first phase of the migration). This condition can lead to unwanted results in the change migration of applications from test to production. For example, defaults of the Page 12

13 new release (test) are not accepted by the old release (production). This situation forces you to complete the release migration in a short period of time. IBM allows for a migration period in which they will charge only for a single release. If you go beyond this period, you must pay for both the old and new release. So many things can push you to complete the upgrade quickly, which is not an easy task when you have many DB2 subsystems. When you rush things or take shortcuts, the possibility of an error increases. Remember - no migration is ever completely transparent to your business; it will cause downtime and can put your data at risk. Applying DB2 maintenance Like every database management system, DB2 needs routine maintenance. IBM provides program temporary fixes (PTFs) to fix errors, patch security breaches, or apply tolerance for an upcoming release. Unfortunately IBM also provides new functionality with some PTFs (often pushed by partner companies). At times, this new functionality has had a negative impact on applications. Declining the new functionality is not an option because PTFs are usually batched and must be processed together. Another side-effect of these fixes is that sometimes you must make a change to your system or application for the fix to work; for example, you many need to rebind packages or run utilities. It is not an option to decline the maintenance entirely, because it is better to fix an error before it hits you. On the other hand, you do not want to be the first to apply new PTFs, unless it s your problem they are fixing, because the fix could cause an error that requires yet another PTF to fix it. Yes, many of us have had this experience. A nice strategy is to apply PTFs that are six months old or older, two times per year. Whatever your strategy is, always read the IBM hold-data (extra information IBM thinks you should read like special instructions) and monitor IBM s hypers (PTFs IBM thinks you should not delay because they address major problems like data corruption or data loss). Release upgrades and DB2 maintenance should always be performed by skilled personnel. Not having enough system programming knowledge is, again, asking for trouble. Availability and performance improvements Keeping your data in a DB2 database means that someone, usually a DBA, monitors its health and performance to ensure that your applications stay in good shape. This is something you might want to document in a Service Level Agreement (SLA). When users/customers complain about response Page 13

14 time and how long it takes to do their work, recent changes to DB2 resources are often the root cause. Sometimes, these changes can even result in downtime. In most companies, business units are not informed of these changes unless a production outage is scheduled. Even then most users don t know what, if any impact there might be to their application and data. When things go wrong, these situations fall into the I never asked for it category. Application performance degradation can be caused by something that has changed like physical characteristics, growth, or new catalog statistics. As a result the access path chosen at bind/rebind time may not provide optimum access to the data. Resolving these issues can be as simple as performing a reorganization or running RUNSTATS, or as intrusive as dropping and recreating a tablespace and its dependencies. Let s look at some of the things we can proactively change to improve access paths and reduce the risks of making a change that delivers poor results. Access paths Plan stability was introduced in DB2 9. With plan stability, DB2 retains backup versions of your programs access paths, providing a safe way to rebind, which is very helpful for fallback. A rebind can improve access paths, but sometimes it does not and this can be a challenge to undo. Plan stability is available for packages only; you can control the level of the stability, which controls the number of copies retained. You will incur additional storage (in the DB2 directory, which often does not have a lot of free space) and additional CPU cycles for a rebind (10% to 40%). How BMC can help When the access path of SQL statements change, the changes can cause performance degradations. The Workload Compare Advisor in BMC SQL Performance for DB2 enables you to compare whole workloads (SQL statements) before a rebind so that you can evaluate the impact of changes before making them effective in the new environment. This comparison can prevent problems when the optimizer chooses a different access path, you make a structure change, object statistics change, or when you migrate to a new version of DB2. Indexes You can influence access path selection by adding an index to a table and running RUNSTATS. For static SQL, you need to rebind the plan or package; for dynamic SQL you need to invalidate the Page 14

15 dynamic statement cache (if used). Adding the new index will not cause any downtime, although the rebind can introduce contention on a busy system. However, it is unwise to play this what if game in a production environment. Virtual indexes In DB2 V8, IBM introduced the concept of virtual indexes. To use this feature, create a virtual index table (SYSIBM.DSN_VIRTUAL_INDEXES) and manually populate it with the appropriate values of the indexes you plan to create. When you execute EXPLAIN, these virtual indexes are considered for the potential access path selection. This allows you try various conditions without risk. Just remember to EXPLAIN all dependent plans and determine any dynamic SQL usage before dropping old indexes or creating new ones. A fix to one package could be a disaster for another. Keep in mind that extra indexes cause extra overhead in insert, update, and delete statements as well in utilities. Changes in indexes can have a negative effect, for example the number of matching columns in the access path decreases because the column order changed. Clustering indexes A DB2 table can only have one clustering index. By default, every table that has one or more indexes has a clustering index. If no indexes are defined as clustering, the REORG utility regards the oldest living index as the clustering index. This is not necessarily the index with the lowest object number (low object numbers get re-used eventually). The REORG utility sorts the data in the tablespace according to the clustering index. During new inserts, DB2 attempts the keep the data in clustering order, but this is possible only when free space permits. Clustering is very important for performance. DB2 9 allows you to alter an index from normal to clustering, but there can be only one clustering index. When you change the clustering index it is likely that the data is not clustered according to the new index. You need to reorganize to reestablish the new clustering order. How BMC can help The BMC SQL Performance for DB2 Workload Index Advisor ensures that existing indexes are optimally structured. It automatically collects and displays actual access counts for each unique SQL statement (table and index, and predicate usage frequencies). For example, you can generate a report that shows which statements access non-indexed columns or how changes to existing indexes affect other SQL statements. Other table and index reports provide quick access to listings of the most used objects based on get page volume or index access ratio. The Workload Index Advisor extends the capability of the Common Explain function within the BMC Availability and performance Page 15

16 improvements solution by comparing access paths after making changes to simulated indexes in a cloned database. A what-if index analysis lets you model changes to indexes. The what-if capability removes the guess work involved when fixing access path problems. The BMC SQL Performance for DB2 Workload Index Advisor provides an automated process to create the best possible indexes for a given SQL workload. Once you have defined the workload (dynamic and static SQL is supported), the Workload Index Advisor will suggest possible new indexes that are currently not defined and will also verify your existing indexes in comparison to the suggested ones. A comprehensive online report provides you a complete overview on all index suggestions ranked by effectiveness. Tablespace The physical data set structure can affect the performance and availability of application data. DB2 9 allows you to create and change three types of tablespaces: partitioned, segmented, and universal (partitioned by range or partitioned by growth). Selecting the appropriate type depends upon the size and growth pattern or clustering requirements of your data. In DB2 9, making a change to the tablespace type always requires a DROP/CREATE scenario. DB2 10 allows you to convert a single table segmented tablespace to a universal tablespace (partitioned by growth) or a normal partitioned tablespace to a universal tablespace (partitioned by range). How BMC can help If you are running DB2 V8 or DB2 9 and would like to migrate to universal tablespace (UTS) without using the UNLOAD/DROP/CREATE/LOAD process and causing a production outage, BMC Recovery Management for DB2 delivers a high speed structure change (HSSC). HSSC transforms the tablespace pages to the new UTS format. This process is much faster than the row level unload/load process. Supported by the unique Online Consistent Copy (a consistent DB2 backup which does not cause any outage no QUIESCE, DRAIN/CLAIM or RO status), HSSC gives you a high availability solution to migrate to UTS, but also to change DSSIZE, SEGSIZE, or the LARGE attribute of a tablespace. Locking Data availability is not only measured by downtime; it is also measured by its concurrency. A transaction that ends in a deadlock or time-out is regarded by the end-user as a failure because the request did not complete. DB2 provides many parameters for utilities and SQL statements and Page 16

17 BIND commands to control the lock duration and sharing of data. Changing concurrency options of an application always requires a rebind of the packages having an impact. Therefore, it is important to carefully select the proper parameters during development and perform tests regarding concurrency. Utilities Making schema changes to tables usually involves one or more DB2 utilities. The parameters used for utility operations affect utility performance and influence data availability. Over the years, IBM has made many utility enhancements that provide online operations and improved performance. But sometimes the online behavior of utilities has unacceptable side effects. The most famous one is probably that IMAGECOPY SHRLEVEL CHANGE cannot be used for RECOVER TOCOPY. DB2 V8 introduced a LOAD SHRLEVEL CHANGE which acts like an SQL INSERT and is very different from SHRLEVEL NONE. Always read the implications of SHRLEVEL CHANGE (the online version) in the utilities manual very carefully. Business application enhancements and fixes Over time most business applications and vendor software products drive change. Behind these changes are requirements to better serve customers, to reduce operating costs, or to take advantage of technology advances and facilities. Responding to these requirements sometimes means modifying or developing application code and altering or creating DB2 objects. Let s reviews some of the many reasons to make changes to your DB2 environment: Application errors (bug fixing) Program logic reflects business rules. When the programs are designed, the analyst or programmer can misinterpret specifications, a specification may be unclear or incomplete, or a programmer can simply make a mistake in implementing the logic. Any of these will result in so-called bugs. Programmers fix the code and test it, and then a new version of the program is ready to go into production, causing a change. This also applies to off-the-shelf application software; the fix and test is done by the software vendor and the result is a new version of the program or a fix (like the PTFs for DB2). Page 17

18 Data pollution and data corruption Data quality is a topic that is heavily discussed. 100% accurate data is probably a utopia, and a certain amount of pollution is always present (for example, end-users discover that they can enter incorrect values on data entry screens and use it to their advantage). When data must be transferred to a data warehouse application, we often talk about cleansing of the data, getting rid of all the anomalies. There may be a need to fix these anomalies in the data in the production environment to avoid propagating bad data throughout the application. Sometimes this can be done in real-time (batch program), but there are times the data needs to be taken offline for this change (using utilities). Of a greater magnitude is data corruption. A program bug can destroy or damage data. Depending on how long this corruption has been going on and how often it occurred, it can be very difficult to fix the bug. It might even involve restoring old copies of the data to further investigate or fix it. Applications are often offline while programmers, DBAs, and business users investigate the damage. Of course, you need an emergency fix for the offending program. How BMC can help DB2 data sometimes gets corrupted by erroneous application logic or human errors. BMC Log Master for DB2 generates SQL to undo the erroneous changes (delete or re-insert the data). You can remove just the transactions in error instead of recovering an entire database, thus saving significant time and money. This can significantly increase the availability of DB2 applications in error situations. New applications and retired applications When new applications are being developed, changes to the program logic and data structures are continuous. Well-designed development and test systems reflect the complexity of the production; therefore, making a data structure change in these systems is as complex as making a data structure change in production. We need a vehicle to ensure that we collect all changes and implement them correctly in the next system (change migration). Often, test and acceptance environments are different from the production environments. Naming conventions and physical storage are not the same (for example, more tables are concentrated in a single tablespace), making it impossible to use the same migration script throughout the development cycle. New applications also have impacts on existing applications when applications interact. Page 18

19 Retirement of applications is something that is usually done the quick and dirty way. In most cases, there is no budget to clean-up old applications so people simply stop the scheduling, dummy out data sets in JCL, or do nothing. You must be very careful with this kind of application pollution. Years later, no one remembers what is what, and no one wants to touch parts of the application because there is a risk they might break something that is important. New or changed business logic Changes to business logic are similar in scope to developing new applications or bug fixing. In fact, these are often bundled into a single (big) change or release level. Compliance Government regulations address privacy issues, data integrity, and security. Certain applications require high levels of accountability. DB2 ensures a compliant and secure infrastructure through trusted context, roles, auditing features, and encryption. Consider making these objects and features an integral part of your application, and make them part of the change management cycle. Compliance, and the associated audit, becomes more and more important. Auditors need tools to compare data structures before and after a change. If your existing change management tools do not have any audit facilities, consider incorporating audit facilities in your applications or change process. The impact and risk of making changes Whenever you implement change in DB2, follow these seven basic tenants of change to ensure that you bring your business applications and/or DB2 infrastructure to the desired post-change state: Communicate to your users and customers before, during, and after you make changes, especially when things are going wrong. Have a documented plan of action that is approved in accordance with your business governance. Ensure that your change incurs the least amount of application downtime as possible or at least meet your production schedule. Preserve the integrity of all data. Adequately test application changes. Page 19

20 Have a back-out plan available just in case. Document what you did, and preserve artifacts for future use and audit. If you leave out or compromise on any of these, you have just increased your risk that something will go wrong. To meet business time frames, a systematic approach is a must. We already touched briefly some potential risks when implementing DB2 changes. Now, let s take a look at a few when things go wrong scenarios. Incorrect implementation It is easy for mistakes to go unnoticed, but the impact of the mistake could become a disaster. Consider a simple-to-fix scenario: You forgot an index. It was dropped and never recreated. After you rebind packages, some transactions have bad response times. You have grumpy end-users and you must debug and fix the situation, but at least no data is lost. But what if you forget a trigger? Because triggers are part of the business logic, that essential logic is now missing. Before the error is detected and fixed, many transactions could have been processed and data might have been corrupted. The mitigation could take days and adversely affect your customers. In scenarios like these, there is no SQL error given or return code set to tell you there is a problem. Simple mistakes can turn into big disasters. Change management tools can prevent mistakes like these and eliminate the risk of costly mistakes. Incorrect backout But even with tools you still have serious risks. Rebinds can create new access paths. How do you ensure that you backed out all the changes from an unsuccessful change implementation (including any updates done to the data during the test)? Let s start with rebind. When you rebind an application, DB2 can opt for a different access path. Many things influence access path selection. Even if you have not made any mistakes in re-creating the objects and have run RUNSTATS correctly, it is possible that DB2 will take a different access path. Because in 99% or more of all the cases the new access path performs as good as or better than the original one, most DBAs consider it an acceptable risk. You can further mitigate this risk with software that provides access path comparisons. Page 20

21 A failed change implementation has a more difficult resolution. If you implement a change and the implementation is rejected during the test or after the application is used in production, then you must back out the entire change completely. Many DBAs don t create a backout scenario in advance. They will try to fix the error on the fly. When this proves to be impossible, they start to back out the change manually without a guarantee that the first attempt will be successful. This is a risky business because it is never practiced. Even when structure changes are undone, in many cases, no one looks after the data. You may have no knowledge of the business processes that affect the data in the application. Has it been corrupted during the test or been updated by users? The only way of undoing everything is to go back to a time when the subsystem and application had a common sync point. The only way to do this quickly is to use snapshot technology for all disks involved. Each type of application changes can have an impact on the business environment in the form of downtime and the potential for data corruption and loss. The more complex the change is, the more risky it becomes. As the implementation scripts needed become more complex, they also become more error prone. If the scripts are written manually, it s easy to make a mistake. The impact of an error can be major. With this same complexity comes the requirement for a higher level of knowledge and skills to prepare and execute changes. Let s look at four application change types: Program-only changes, no DB2 data structure changes required, such as SQL statements and business logic Data structure changes with minimal impact, such as adding a new index Data structure changes using online change, such as altering columns The high-impact change using unload, drop, create, load, and so on, such as deleting columns Why is it so hard? The book Database Administration: The Complete Guide to Practices and Procedures by Craig Mullins (ISBN ) is a masterpiece on database administration and a must-have for every DBA. Craig writes, The DBA is the custodian of database changes. Usually, the DBA is not the one to request a change; that is typically done by the application owner or business user. But there are Page 21