Improving Information Delivery and Reducing Cost of Ownership in an IBM Data Environment

Transcription

1 Improving Information Delivery and Reducing Cost of Ownership in an IBM Data Environment A Guide to Integration between IBM DB2 for i and MySQL Enterprise July 2009

2 Contents Executive Summary 3 The High Cost of Information Delivery 4 Flexibility at a Lower TCO 4 Approaches to Integrating Data between IBM DB2 for i and MySQL 6 MySQL as a Low Cost Alternative to IBM DB2 7 HiT Software s DBMoto Data Replication Solution 8 DBMoto: Step-by-Step 10 Real-World Case Study of IBM DB2 for i and MySQL Integration 14 HiT Software, Inc. 15 Integration Between IBM DB2 for i and MySQL Enterprise 2

3 Executive Summary Organizations today depend on data from all facets of their business for decision-making, operational management and core day-to-day transactions. IBM i (AS400) systems are a mainstay in medium to large organizations and IBM databases are often used as the core repository for production, operational and financial data. While IBM i systems are historically very reliable, they are expensive and require specific expertise for implementation and management. This becomes a challenge when organizations need priority access to the data stored in their IBM systems, but do not want to affect the workload and processing for which the IBM system is responsible, and do not have enough trained expertise in-house to customize required data access to the IBM databases. This paper offers guidance on the use of a cost-effective, alternative database option and an adaptable data integration strategy, to make IBM i data readily available for business requirements. Taking advantage of HiT Software s DBMoto data replication solution to integrate data between IBM i and MySQL Enterprise, businesses can reduce cost and time in managing large-scale, heterogeneous database environments, while leveraging the investments they ve made in database and hardware systems. Integration Between IBM DB2 for i and MySQL Enterprise 3

4 The High Cost of Information Delivery A fundamental aspect of business administration is the coordination and management of corporate data. Financial information, sales orders, customer data, parts/supplies, operations, purchasing and human resources management all depend on the management of data and, for most enterprises, data management has become mission-critical. With the extensive development of custom and packaged applications, data types and definitions are not universal. In addition to many online transaction processing (OLTP) databases, the past decade has produced a number of analytic databases, i.e., data marts and data warehouses, again expanding data types and formats. The issue of data types becomes a bottleneck when organizations need to have a holistic view of corporate facts and figures, and requires the integration of data from multiple and disparate databases which may use distinct formats. While there are many choices for enterprise database management systems, here we consider the advantages of integrating data in an organization which has already made an investment in IBM i (AS400), and wants to leverage that investment while exploring ways to make its business data available to its employees in the most cost-efficient way. As a SQL database, IBM DB2 for i has much in common with other SQL databases, sharing a SQL query language and basic relational structure. It has become a highly dependable workhorse for the heavy loads and stress of a production system, and is used in many 24x7 operations, such as manufacturing and processing. However, in current business conditions, IT organizations find that the IBM i environment is not always able to meet their needs for cost-effective, versatile access to data. Good IBM i database administrators, with considerable specialized experience in the management and understanding of the underlying IBM i operating system, are costly, overloaded with administrative tasks, and hard to find. As capable as the IBM i system is for production reasons, its cost can exceed tight IT budgets for new development. Furthermore, as new departments and groups need access to the data produced by the IBM system, the IBM i cannot successfully manage its core job as the data repository and, at the same time, make itself available for answering multiple queries without a severe impact on the performance of its data processing. Flexibility at a Lower TCO The solution to this problem is the use of real-time incremental data replication to create and maintain accurate replicas of core data (either groups of tables or entire databases.) This approach provides a seamless way for corporate users to access the same integral information, without impacting the IBM production system. With recent technology advances in open source software, businesses are beginning to make long-term investments in systems that take advantage of the cost-savings and ubiquity of open source solutions and standards. While many open source solutions are overlooked for lacking the necessary infrastructure and support that mainstream commercial products offer, a few products stand out for their ability to offer businesses a market-leading, proven technology, as well as a plethora of options that support the widest spectrum of users from individuals to the largest global enterprises. Integration Between IBM DB2 for i and MySQL Enterprise 4

5 Figure 1: HiT Software s DBMoto replication solution provides a complete approach to integrating data between IBM DB2 for i and MySQL. Businesses can reduce cost and time in managing large scale, heterogeneous database environments, while leveraging investments made in database and hardware systems. One of these open source technologies is Sun s MySQL relational database management system (RDBMS). With the MySQL Enterprise offering, MySQL takes its leading enterprise database system and adds a number of IT-critical components such as service, support and extensible add-ons that place the system squarely in the camp of the largest commercial database systems available today. According to Gartner 1, Linux has become a leading database management platform, and has surpassed $2.7 billion in 2007 with more than 20% share of the market as it continues to grow. MySQL's market share among developers has surged to 40%, gaining 25% in the last two years, according to Evans Data Corporation 2. Components in the architecture described above: IBM DB2 for i The native database for IBM i MySQL Enterprise Sun s commercial offering for an enterprise relational database management system HiT Software DBMoto HiT Software s data replication product for bi directional replication between DB2 for i and MySQL (includes HiT Software s connectivity providers) HiT Software Ritmo for IBM i HiT Software s application access provider for.net applications requiring DB2 for i data, where synchronization is not required 1 Gartner, How Open Source Impacts the RDBMS Forecast, Evans Data Corp, Evans Data Survey Results, 2007 Integration Between IBM DB2 for i and MySQL Enterprise 5

6 Today, we often see MySQL in environments where IT projects require database prototyping and testing the value of open source makes it an ideal, cost-effective testing database that can mirror the activities of a comparable commercial database. However, there is a much greater value in MySQL as a standardized corporate database system that resides in parallel with commercial systems, such as Oracle, Microsoft SQL Server, and IBM DB2 for i, Linux, UNIX, Windows and mainframe. MySQL Enterprise is fast becoming an attractive option to cost-effectively and efficiently relieve stress on the incumbent production database through a proven replication system that synchronizes data between the IBM DB2 database and a MySQL copy. In this way, connection to the IBM system is significantly minimized, while MySQL delivers up-to-date and accurate data in a cost-effective way a requirement for smart businesses that depend on access to operational and analytic data for reporting and management. Key Benefits from a Replication Scenario with IBM i and MySQL Enterprise Faster time-to-analysis Reduced stress on the production database Simplicity, adaptability and ease-of-use Lower Total Cost of Ownership, with adoption of an open source database and cost-effective data replication technology Approaches to Integrating Data between IBM DB2 for i and MySQL CUSTOM CODING WITH DATABASE TOOLS Custom coding has the following limitations: Requires a significant investment of time and special IT programming expertise to create a customized solution for integration that then becomes expensive to implement and maintain. The solution is typically developed to solve a specific problem and therefore lacks flexibility and scalability. The solution will typically open an independent connection to the source database every time a request is made, causing potential downtime and bottlenecks on the source database. With the passage of time, costs for maintaining and updating the application can be expensive due to ongoing developments in the database environment and changes in staff. Documentation management, and continuity in training and customer support, are typically lacking. Integration Between IBM DB2 for i and MySQL Enterprise 6

7 MYSQL REPLICATION SERVICES MySQL offers support for migration from any database to MySQL, through the use of a JDBC driver. However, this approach does not support key criteria in data integration: Transactional replications from IBM DB2 to MySQL are not supported. Transactional replications utilize the ability to read IBM journals, allowing changed data capture which significantly reduces performance impact and replication time. MySQL offers only snapshot replications, which require a full copy to be made every time a change is detected. Lack of support for changed data capture and for real-time synchronization Realizing the importance of heterogeneous data replication, MySQL recommends the use of HiT Software s DBMoto data replication product for any data replication job involving enterprise data replication among disparate databases. USING A THIRD-PARTY SOLUTION The advantages of using a product such as HiT Software s DBMoto to replicate data between IBM i and MySQL are: DBMoto keeps track of core database changes by reading IBM i journals to identify INSERT, UPDATE and DELETE operations. Changes are then propagated to the MySQL database based on a schedule determined by the user (including real-time), significantly improving replication speed. The process of configuring and running the replication is flexible and can be used for many different relational databases not just IBM DB2 and MySQL. DBMoto provides real-time, bi-directional synchronization of data between major databases Very small footprint on incumbent systems No data loss during replication tasks or interruptions An intuitive graphical user interface provides easy replication configuration, and no programming is required MySQL and IBM i servers remain secure because DBMoto is completely non-invasive MySQL as a Low Cost Alternative to IBM DB2 Enterprises can reduce the total cost of ownership of their database infrastructure by taking advantage of the MySQL Enterprise database system in a data replication scenario with IBM DB2 for i. As an open source database, MySQL Enterprise eliminates upfront licensing costs, eliminating what could have been additional capital expenditures with the expansion of another IBM i system. Integration Between IBM DB2 for i and MySQL Enterprise 7

8 An IBM i system requires heavy administration and training, while MySQL, running on non-specialized operating systems UNIX, Linux and Windows, does not require specialized knowledge beyond typical database administration. Application development is easier to accomplish in the widely-used operating system environments supported by MySQL. Developers are easier to find and languages such as C#, Ruby, Groovy, AJAX, and Java offer efficient data adapters or drivers to access data. MySQL offers exceptional quick-start capabilities with the average time from software download to installation completion being less than fifteen minutes. This rule holds true whether the platform is Microsoft Windows, Linux, Macintosh, or UNIX. Once installed, self-management features like automatic space expansion, auto-restart, and dynamic configuration changes take much of the burden off database administrators. MySQL also provides a complete suite of graphical management and migration tools that allow a DBA to manage, troubleshoot, and control the operation of many MySQL servers from a single workstation. Although MySQL is offered as open source software, complete around-the-clock support and indemnification are available through MySQL Network. With low-cost commodity hardware, reliability and easy maintainability of MySQL, database administrators do not need to waste time troubleshooting performance or downtime issues, but instead can concentrate on making a positive impact on higher level tasks that involve the business side of data. HiT Software s DBMoto Data Replication Solution IDEAL FOR REPLICATING DATA FROM IBM DB2 for i TO MYSQL ENTERPRISE The DBMoto Solution HiT Software s DBMoto is the first replication product specifically designed to natively support IBM DB2 for i. DBMoto surpasses scheduled replications by supporting real-time incremental data replication, thereby creating an effective and up-to-the-minute channel between corporate data and enterprise applications. With DBMoto, modifications to data on IBM i are instantly reflected in MySQL. DBMoto is highly flexible and can be customized to your environment. In addition, the intuitive wizard-based interface makes it easy to set up replication rules and schedules in minutes. Journals and Logs The DB2 journal provides a native tool for intercepting changes that have occurred in DB2 databases. Each INSERT, UPDATE and DELETE operation is recorded in the journal. Using HiT Software s proven DB2 connectivity technology, DBMoto efficiently accesses and reads the journals in order to capture data changes. DBMoto natively reads the MySQL transaction logs, for replication between IBM DB2 for i and MySQL databases. Integration Between IBM DB2 for i and MySQL Enterprise 8

9 DBMOTO MAIN FEATURES Snapshot replication (Refresh) All selected records are scanned and replicated. Real-time incremental (Mirroring) Any changes in the DB2 journal (transaction log) are captured and replicated from source to target. Synchronization Real-time incremental replication where bi-directional changes are made to both source and target Easy to use The entire replication process is configured using wizards. Replications can be scheduled, executed and monitored using the Enterprise Manager graphical user interface. IBM i Benefits Independent from IBM i - no programming is required on the IBM server and there is no need for proprietary naming or reserved fields in DB2. Small footprint on IBM systems. Accessible The Enterprise Manager Console of DBMoto can be operated on a remote system from the source and target databases to facilitate configuration and management of replications. Customizable Using Microsoft s VB.NET technology and replication events generated by DBMoto, you can write scripts, or insert pre-written scripts to customize the replication process. Flexible In addition to DB2 and MySQL, DBMoto can be used for replications between all major relational databases. For a complete list, check HiT Software s website at Cost-effective SNAPSHOT REPLICATION DBMoto s snapshot replication reads all data (as specified by mapping and filtering) from the source table and transfers it to the target table. Snapshot replication can be launched directly from a menu, or scheduled at intervals specified by the user. Typically, snapshot replication is run once during the initial phase of the replication process to establish the initial data set on the target. It could then be followed by incremental replication to capture subsequent data changes. INCREMENTAL REPLICATION DBMoto s real-time incremental replication examines the DB2 journal (or the MySQL transaction log) for database changes since the last replication, and then replicates only those changes to the target database. The interval between each log read is typically set at one minute s length, but this period can be adjusted by the user. Replicating only those records that have changed allows: Real-time replication, as less time is required to process changes only Minimal workload on the IBM server and the server running MySQL Integration Between IBM DB2 for i and MySQL Enterprise 9

10 Selecting Data for Replication Replication, either snapshot or incremental, applies to an entire table or to a part of the table as defined by the user. For each replication, data is selected in the Replication wizard by: Mapping the columns to replicate Filtering data using expressions Writing scripts to filter or modify data during replication Examples of column mapping: Code and Description columns only Social Security, telephone and fax columns only Code, Password and columns only Examples of filter criteria: Timestamp Clients with online services only European distributors only DBMoto: Step-by-Step Replications can be quickly configured in the DBMoto Enterprise Manager console following a straightforward installation via a downloadable Microsoft Windows setup file. INTUITIVE REPLICATION CONFIGURATION Step 1: Start the DBMoto Enterprise Manager: Integration Between IBM DB2 for i and MySQL Enterprise 10

11 Step 2: Expand the metadata node to reveal sources and targets discovered automatically by DBMoto. Click on Sources to begin configuring the connection to the source database. Step 3: Configure the connection to the source database using drivers provided by DBMoto, or from the database vendor. Integration Between IBM DB2 for i and MySQL Enterprise 11

12 Step 4: Configure the connection to the target database. Step 5: Create the target tables as needed. Integration Between IBM DB2 for i and MySQL Enterprise 12

13 Step 6: Configure the replication a. Select the preferred replication mode: Refresh (snapshot), Mirroring (changed data capture); or Synchronization (bidirectional changed data capture) b. DBMoto provides easy wizard assistance in setting up complex replications such as multiple replications between your database pairs. Plan your replication scheduling and set start times. Note that replications can be run as often as within seconds of each other for real-time updates. Set the replication details (tables, fields and transformation functions) Define scripts associated with the replication Run the replication ACCESSIBLE, CUSTOMIZABLE AND FLEXIBLE Scripting Environment DBMoto is the only real-time replication product that provides a pre-compiled scripting environment fpr optimum performance. Users can define functions, procedures and variables from within DBMoto. Functions can be used in the following three ways: Mapping expressions Mapping rules Replication events While capturing data changes and applying the default mapping, DBMoto provides scripting events that can be used to manage complex transformations and mapping rules. DBMoto offers built-in functions as well as the ability to create user-defined custom functions to manage enterprise data as needed. Integration Between IBM DB2 for i and MySQL Enterprise 13

14 Real-World Case Study of IBM DB2 for i and MySQL Integration HEALTHCARE Oregon Health Systems (OHS), a leading managed care organization, has a provider network of approximately 1500 physicians. OHS sophisticated medical management system serves thousands of workers in its region for workers compensation plans and care. OHS, as an independent managed care organization (MCO) of physician and clinical treatment for occupational injuries and illness, is responsible for managing the patient history from enrollment through resolution, identifying key events and communicating on a regular basis with physicians and claims adjusters. First certified in limited areas in 1991, OHS now offers services to workers throughout the entire state of Oregon. It is the only MCO certified for the entire state. OHS also offers managed care services in Montana as Montana Health Systems, and other case management and cost containment services in several other western states. MCOs are utilized by workers' compensation insurers and self-insured employers to provide injured workers appropriate, cost-effective medical treatment and disability management, and to ensure the claims process and medical care are moving forward. Oregon Health Systems requires the ability to integrate patient, insurance, treatment and disability information in their process of overseeing worker care. According to Ryan Hopman, Systems Integrator and Consultant, Vertigo Group LLC 3, OHS faced a common situation where they had to integrate data from various formats and databases. As a management service, they have to be able to see data holistically in order to determine optimized treatment and physician recommendations, as well as track the patient history. OHS critical requirement was to be able to use IBM DB2 data in their provider management system, and key considerations were time-to-value and a streamlined and efficient IT system. Using HiT Software s DBMoto data replication capabilities, we were able to quickly and easily integrate data from MySQL, IBM and other systems in our solution for OHS, as well as reduce development time, added Hopman. OHS now has a reliable solution that can scale to fit their growth and expansion plans. 3 Vertigo Group is a business management and technology consulting firm located in Portland, Oregon. The company specializes in low-risk, high-return consulting with customized solutions. Vertigo Group solutions are based on extensive industry expertise, broad and evolving service offerings, expertise in business transformation outsourcing, a history of technology innovation and implementation including research and development, and a proven and experienced management team. Vertigo Group is committed to long-term relationships with its clients. Integration Between IBM DB2 for i and MySQL Enterprise 14

15 HiT Software, Inc. HiT Software is a leader in database integration software products. HiT Software products provide access to critical data, enable data availability and offer programming-free data integration across enterprise systems. HiT Software s standards-based products include DBMoto, Allora and DB2 Connectivity. HiT Software products perform real-time, bi-directional replication between all major databases; execute real-time, bi-directional transformations between XML and all major databases; and connect applications to IBM DB2 databases via.net, OLE DB, ODBC and JDBC standards. Founded in 1994, HiT Software is relied upon by thousands of organizations around the globe. HiT Software products are sold and supported worldwide by the HiT Software direct sales team, international subsidiaries, and authorized channel partners. Additional information is available at info@hitsw.com HiT Software, Inc. All rights reserved. All trademarks or registered trademarks herein are the property of their respective owners _a Integration Between IBM DB2 for i and MySQL Enterprise 15