White Paper Success Keys For Migrating From Sybase ASE to DB2 UDB Part 1: Data Migration The purpose of this paper is to describe the key success factors that help migrate the Sybase Database schema, underlying data and the security permissions to DB2 UDB. It clearly shows that the success depends not only on the strategy and the approach but also on the usage of appropriate tools/scripts and the adoption of best practices during the execution.
About the Authors Vairan Rajendran Vairan Rajendran is a senior consultant working with the GCP Infrastructure Consulting Group. He has more than 15 years of experience, covering a wide range of areas including Performance, Capacity & Availability Management, IT Infrastructure Blue-printing, Technology Refresh, IT Architecture and Transformation solutions. Sajad Javid Sajad Javid is a consultant with the GCP Infrastructure Consulting Group. He has more than 12 years of experience, covering a wide range of areas including Performance, Capacity & Availability Management, Architecture solutions, System Integration and Technology Refresh Programs Asherin Rahim Asherin Rahim is an assistant consultant working with the GCP Infrastructure Consulting Group. She has more than 9 years of experience covering a wide range of areas including Performance, Scalability and Technology Refresh Programs. 2
Abbreviations and Acronyms The following table lists the abbreviations used in this document. Abbreviation Description DB2 IBM Database 2 DBMS Database Management System DDL Data Definition Language ETL Extract, Transform & Load GCP Global Consulting Practice IDMT IBM Data Movement Tool MBCS Multi Byte Character Set MTK IBM Migration Tool Kit PROD Production SBCS Single Byte Character Set TCS Tata Consultancy Services UAT User Acceptance Testing UDB Universal Database 3
Table of Contents 1. Why Migrate From Sybase? 5 2. Overall Data Migration Approach 5 3. Success Key 1: Build An Accurate Source Database Inventory 7 4. Success Key 2: Build A Comprehensive Pattern Repository 7 5. Success Key 3: Use A Precise Schema Migration Strategy 8 6. Success Key 4: Use A Precise Data Migration Strategy 11 7. Success Key 5: Choose Right Tools For The Migration 12 8. Success Key 6: Use An Appropriate Security Migration Strategy 13 9. Other Considerations 14 10. Conclusion 15 4
Why Migrate from Sybase? When we stepped into Mike's New Jersey office, he looked a little pensive and withdrawn. There were clearly some pressing challenges that he was facing. After discussing the issue for a few minutes, we learnt that his current database portfolio was the main cause of concern. The top management has asked Mike to rationalise and optimise his database portfolio so that the total cost of ownership (TCO) could be reduced, and he turned to us for help. TCS took on the challenge and our first step was an assessment of the database portfolio. One of the key findings was that Mike had the enterprise-wide license to use DB2 UDB. The Sybase ASE was being used for a small set of applications which had very few reporting jobs. The usage analysis of these Sybase reports revealed that some of these were not used at all. Hence, we suggested moving to DB2 UDB and merging the Sybase reporting functionality to one of the existing applications. Though Mike was happy that this would reduce TCO considerably by eliminating the Sybase ASE license and support costs, he was concerned about implementing the recommendation. This paper provides pointers on how to migrate smoothly from Sybase to DB2 UDB. In a multi-dbms environment, there are other key factors which may trigger similar DBMS consolidation projects: High operational costs to business value ratio Unsupported versions of Sybase Issues with upgrade from the current version to higher versions High license cost for migration from the current version to higher versions Minimal skill availability Technology Adaptation/Standardisation to a common database across the enterprise Overall Data Migration Approach In any migration project, while a good strategy plays a major role, the application of the strategy is equally important. The right approach enables the migration team to get the answers to the following key questions and guides them in the deployment of the strategy: What are the key elements that must be included as part of the migration? What strategy should be applied during migration for each of these key elements? What is the target state for these key elements? What is the impact of moving these elements to the target state? What are the key enablers for the smooth migration of these elements? What are the cost implications of this migration? What validation mechanisms/strategies are available to verify the migration? What are the resource requirements for this migration? The answers to the above key questions will help plan a winning approach for the migration. 5
The following diagram describes a typical Sybase to DB2 UDB data migration approach involving Schema migration, Data migration and Security migration. Schema Migration Security Migration Data Migration Extract Metadata Extract Security Data Generate Scripts Extract Metadata Create Metadata inventory Extract Logins / Users / Groups Extract Object Authorizations Use 3 rd Party Tools to Generate Scripts Analyze & Fix Tool Errors Source Data Scanning Create Pattern repository Analyze Process Changes Code page considerations Scan for special characters Generate and Run Scripts Generate and Run Scripts Execute Scripts CLOB considerations Use 3 rd Party tools to generate scripts Scripts for Users, Groups in AIX Run Load / Import Scripts to Load Data Data Size analysis Run Scripts to Generate DDL Scripts for Object Authorizations in DB2 Analyze & Fix Errors Run DDL in Target DB2 Server Run Scripts both in AIX and DB2 Validate Schema Migration Validate Security Migration Validate Data Migration Extract Target Metadata Validate Users and Groups Run Row count and Audit checks Validate against Patterns Validate object authorizations Carryout Full data Compare Figure 1: Overall Data Migration Approach The Schema and Data Migration is carried out in the following phases: 1. Schema migration This is the first step and involves the migration of the database structure and objects, along with the migration of security permission and authorisation required for the users and groups to login to the database. 2. Data Migration This is the second step where the data from the source Sybase tables is extracted using a tool/script and loaded on to the target DB2 UDB tables in single or multiple steps, depending upon the complexity and size of the tables. 6
Success Key 1: Build an Accurate Source Database Inventory The first and most important factor for the success of the migration is the creation of an accurate inventory consisting of database instances, schemas, database objects, data types used, users and groups. These details have to be collected for each of the environments (Development, Test, and UAT), along with the size of the data and the log files, in order to create an effective migration strategy. The following details are collected as part of the inventory: List of environments List of database instances List of database schemas List of database objects such as tables, aliases, views, indexes, triggers, constraints, procedures, rules, and so on List of data types used Size of each database instance List of users and groups At the basic level, the above inventory will help plan for the resources required for the migration. Hence, it is important to have a complete inventory to arrive at the resources and the cost elements for the migration. Success Key 2: Build a Comprehensive Pattern Repository The second important factor is the creation of a comprehensive pattern repository. This repository should have both the source data type patterns and the target data type patterns, and this involves the identification of: the source data type patterns the equivalent target data type patterns the impacted patterns in the source that need to be changed during migration 7
Source and Target Data Types The following table provides a snapshot of the various Sybase data types and the equivalent DB2 UDB data types: SYBASE CHAR(N) VARCHAR(N) NCHAR(N) NVARCHAR(N) TEXT SYSNAME SMALLDATETIME DATETIME DATE TIMESTAMP TINYINT SMALLINT INT DB2 UDB CHAR(N), VARCHAR(N) VARCHAR(N) GRAPHIC(N), VARGRAPHIC(N) VARGRAPHIC(N) CLOB(N) VARCHAR(30) TIMESTAMP TIMESTAMP DATE TIMESTAMP SMALLINT SMALLINT INT SYBASE DB2 UDB NUMERIC(P,S) NUMERIC(P,S) DECIMAL(P,S) DECIMAL(P,S) REAL REAL FLOAT FLOAT DOUBLE PRECISION DOUBLE BINARY CHARACTER(N) For Bit Data VARCHAR(N) For Bit Data VARBINARY VARCHAR(N) For Bit Data IMAGE BLOB(N) BIT SMALLINT SMALLMONEY User Defined Type (UDT) with DECIMAL (10,4) MONEY User Defined Type (UDT) with DECIMAL (19,4) IDENTITY IDENTITY Table 1 Data Type Impact Analysis Identify all the impacted data types by comparing the source and the target data types. Identify the occurrence of the impacted data types across the databases. Based on the quantum of the impact, decide on the data type conversion process either automated or manual. Pattern Repository Build a pattern repository to store both the source and the target data type patterns. The source data types from Sybase and the target DB2 UDB data types, along with the transformation scenarios, need to be stored in a pattern repository. Utilise this repository while validating the schema migration process, and update it on a regular basis to account for the changes discovered during the validation process. Success Key 3: Use a Precise Schema Migration Strategy The third important factor for the success of the migration is the schema migration strategy. Knowing the answers to the following questions will help the migration team build a successful schema migration strategy: Do we have an accurate inventory of the schema/database objects (tables, aliases, indices, constraints, and so on)? How will the schema be migrated? For example, using 3rd party tools, scripts, and so on. What is the validation strategy for the migrated schema? 8
Schema migration refers to the set of activities that must be performed in order to migrate a portion of the underlying data structure/metadata of the Sybase database system. These activities include migrating tables, aliases, associated indices and constraints from Sybase Database to DB2 UDB. Note: The other database objects including stored procedures, triggers and views will be migrated as a part of the application migration which is detailed in the next white paper, Success Keys for Sybase to UDB Part 2: Application Migration. Sybase Scan for Special Characters (based on code page) Extract Source DDL Identify Column Length Changes Generate Custom Scripts Use 3rd party tool for Schema Migration Script Generation Execute Scripts to Generate DB2 DDL DDL Post Processing Table Space Mapping Create Schema in DB2 Database Pattern Repository Validate Migration Figure 2: Schema Migration Db2 UDB The following points should be considered while migrating schema from Sybase to DB2 UDB: In DB2, all column names are capitalised by default and are case insensitive. This will result in all column names being in upper case in DB2 UDB. This will not affect the way they are accessed in applications. While loading tables with IDENTITY columns, default identity processing will be bypassed in the DB2 LOAD process to properly migrate identity values from Sybase to DB2 UDB. Code Page Changes DB2 UDB mostly uses UTF-8 Unicode for encoding data. Sybase, version 12.5.1 onwards, supports the Unicode character set. However, some of the old Sybase installations still use single-byte character set (SBCS) code pages such as ISO-8859-1, CP850, among others, to encode data. 9
Unicode supports multi-byte character sets (MBCS). Moving data from a SBCS to a MBCS character set can cause data truncation in the target environment (depending upon the underlying data and the size of the table column), unless the target data type is handled appropriately. So, during the schema migration process, a special character scan should be performed to identify potential truncation candidates and configure the data tools to transform the data to Unicode. A strategy needs to be applied to avoid truncation. For example, DB2 data types NCHAR and NVARCHAR can properly take Multi-byte Unicode data without spanning multiple positions and can therefore avoid data truncation. So, one strategy could be to replace all character columns that have special characters occupying multibytes in DB2 to NCHAR or NVARCHAR (instead for CHAR and VARCHAR respectively). Schema Validation It is important to verify and validate the schema migration before proceeding further, since data migration depends heavily on the metadata. By validating the schema migration in the development/test databases, one can become familiar with the migration tools and the process itself, and this helps to understand the following: Which special cases apply to the environment and which migration approach will be most efficient in the current context? Where are the log files saved and how can they be read? What information do they provide? Which techniques may be used to mitigate downtime? How can problems that occur be dealt with? A combination of DB Solo (a third party tool) and Custom scripts can be used to ensure that the schema conversion is complete. DB Solo produces a report with the list of changes (all objects) between the source and target database. Custombuilt scripts can be used to compare the mismatches with the data type change patterns to flag changes as genuine or in-error for validation. DML Validation Post schema migration, all DML operations should be performed on a set of migrated tables to ensure normal SQL processing (INSERT, UPDATE, SELECT and DELETE). 10
Success Key 4: Use a Precise Data Migration Strategy The fourth important factor is the data migration strategy. For this, it is essential to know the answers to the following questions: How will the data be migrated to the target platform? By what means will the data be migrated using 3rd party tools, scripts, and so on? What is the validation strategy for the migrated data? Data Migration Strategy A Sybase to DB2 UDB data migration strategy is depicted in the following diagram. It involves following steps: Extracting the source data from the Sybase database Loading the extracted data into the target DB2 database The migration team usually uses third party tools to perform the ETL process and any post processing steps required on the migrated data. Use 3rd party tool for Data migration script generation Sybase Execute Scripts to extract data from Sybase Custom scripts from Schema Migration Load data to Db2 Validate Data Migration Row count Audit checks Full data compare Db2 UDB Figure 3: Data Migration 11
Verification and Validation For verification and validation of the data migration from Sybase to DB2, a combination of DB Solo and custom built scripts may be employed to ensure that the data migration is correct and complete. Data comparison can be a time consuming process, especially for tables with a large amount of data (more than 10 million). One strategy could be to perform the full data comparison in test stages till the dry run for sign-off. During the actual migration (in PROD environment), a subset of data can be compared to ensure correctness. DB Solo can aid in comparing data from a set of tables and report on the differences to make the necessary modifications (if required) SELECT Queries can be run against tables containing special characters to validate the success of the data migration Post data-load into DB2 UDB, Indices may need to be rebuilt and statistics need to be updated for best performance Post migration backups at multiple levels (post schema and post data) will be required to enable better control of fallback levels during checkout. Success Key 5: Choose Right Tools for the Migration The fifth important factor for the success of the migration is the selection of a suitable tool for the migration. The key-base criteria for the tool selection include the following: Source data extraction capabilities Source data transformation capabilities Data loading capabilities Data staging capabilities Exception handling Schema and Data migration from Sybase Database to DB2 UDB may be accelerated using a combination of various tools that are available with third party vendors. Some of these tools are supplied along with the DBMS product itself. The following sub-sections describe the key features of various tools that may be used in the migration. Migration Tools IBM Migration Tool Kit IBM Migration Tool Kit (MTK) is a free utility from IBM to migrate a wide range of databases to DB2 UDB. MTK is an end-to-end migration solution and can convert database schema, database objects and data from other database systems to DB2 UDB. MTK is Java based and is available on both Windows and UNIX based platforms. MTK can connect to source databases, extract source metadata and create Scripts for data migration. 12
Ispirer SQLWays SQLWays is a commercial product from Ispirer Systems, that aids in the conversion of database objects, including application codes, from one database system to another. SQLWays is a licensed product, and provides an end-toend solution for migration with a high degree of conversion. SQLWays can connect to source databases using different DB protocols, and can extract metadata and table data. IBM Data Movement Tool IBM Data Movement Tool (IDMT) is a free tool available from IBM that can aid metadata and data migration. It is Java based, and can run on a variety of platforms. IDMT can connect to source databases, extract source metadata and create Scripts for data migration. IDMT has two distinct options to extract and load data during migration: Intermediate File-based Operation - Data is unloaded from Sybase in to a intermediate file and then loaded into Db2 Unix Pipe-based Operation: Data unload and load happens using a single Unix pipe Success Key 6: Use an appropriate Security Migration Strategy The sixth important factor is the security migration strategy. In Sybase, users are locally setup on the DBMS and the authorisations are given to these users and groups. But in DB2 UDB, users are created in the underlying operating system and DB2 UDB makes use of these operating system services for authentication. The following diagram describes the strategy for migrating the users, groups and authorisations from the Sybase system to DB2 UDB on the AIX platform. Sybase Extract User List from Sysusers - Sysalternates Extract Groups and Roles Extract Object Permissions Create Users / groups in AIX Validate Security Migration Apply Object Permissions AIX Server Db2 UDB Figure 4: Security Migration 13
The following items should be considered while migrating the security permissions from Sybase to DB2 UDB: In DB2, authentication is handled by the underlying operating system as against the database system in Sybase. Sybase logins and users should be cleaned up before they are migrated to AIX. Sybase database passwords need not be migrated because the DB2 UDB operating system services handles the authentication. Users should use their AIX passwords to log on to DB2 UDB. Sybase database object level permissions must be split-up and only the table level accesses will be migrated to the target system. The remaining accesses will be handled during the application migration process. During the production implementation, all the database object level authorisations must be applied only after the application database components are migrated, in a single step. Security Validation Typically, the following metrics are collected to validate the correctness of the security migration. Audit Metrics User Count Group Count User Authorisation Object Authorisation Description Comparison of count of user Logins on Sybase against AIX Comparison of count of Groups on Sybase against AIX Count of DB users against Connect access on DB2 Comparison of object authorisation from Sybase against DB2 Table 2 Other Considerations Though this part covers most of the aspects of the Sybase to DB2 UDB data migration, it is also important to follow some of the standard data migration best practices during the execution, such as: To migrate database schema and stored procedures, use the IBM MTK Tool which can greatly reduce the manual effort To generate the DDLs for the target databases, use the Erwin Data Modeler which can greatly reduce the manual effort Use scripts to automate the data migration process, and thereby reduce manual effort Use source code dump to cross verify the inventory and the analysis results 14
Conclusion A combination of business needs and technology investments are driving the Sybase exit migrations. Sybase to DB2 UDB migration may seem easy at first, but a cautious approach should be adopted by building a winning data migration strategy, creating a detailed migration approach and using tools/scripts wherever possible. The following success factors should be considered while carrying-out a Sybase to DB2 UDB Data migration programs: Building an accurate source database inventory Building a comprehensive pattern repository Using a precise schema migration strategy Using a precise data migration strategy Choosing the right tools for the migration Using an appropriate security migration strategy 15
About TCS Global Consulting Practice TCS Global Consulting Practice (GCP) is a key component in how TCS delivers additional value to clients. Using our collective industry insight, technology expertise, and consulting know-how, we partner with enterprises worldwide to deliver integrated end-to-end IT enabled business transformation services. By tapping our worldwide pool of resources - onsite, offshore and nearshore, our high caliber consultants leverage solution accelerators and practice capabilities, balanced with our knowledge of local market demands, to enable enterprises to effectively meet their business goals. GCP spearheads TCS' consulting capacity with consultants located in North America, UK, Europe, Asia Pacific, India, Ibero-America and Australia. Contact For more information about TCS' consulting services, email us at global.consulting@tcs.com, or visit www.tcs.com/consulting Subscribe to TCS White Papers TCS.com RSS: http://www.tcs.com/rss_feeds/pages/feed.aspx?f=w Feedburner: http://feeds2.feedburner.com/tcswhitepapers About Tata Consultancy Services Ltd (TCS) Tata Consultancy Services is an IT services, consulting and business solutions organization that delivers real results to global business, ensuring a level of certainty no other firm can match. TCS offers a consulting-led, integrated portfolio of IT and IT-enabled infrastructure, engineering TM and assurance services. This is delivered through its unique Global Network Delivery Model, recognized as the benchmark of excellence in software development. A part of the Tata Group, India s largest industrial conglomerate, TCS has a global footprint and is listed on the National Stock Exchange and Bombay Stock Exchange in India. For more information, visit us at www.tcs.com IT Services Business Solutions Outsourcing All content / information present here is the exclusive property of Tata Consultancy Services Limited (TCS). The content / information contained here is correct at the time of publishing. No material from here may be copied, modified, reproduced, republished, uploaded, transmitted, posted or distributed in any form without prior written permission from TCS. Unauthorized use of the content / information appearing here may violate copyright, trademark and other applicable laws, and could result in criminal or civil penalties. Copyright 2012 Tata Consultancy Services Limited TCS Design Services M 05 12