Data Integration Cost Considerations

Transcription

1 WHITE PAPER Data Integration Cost Considerations Tips for Real World Implementations

2 Table of Contents PAGE 2 Executive Summary 2 One Database, One Application, One Solution? 3 Data Integration Scenarios 4 Data Integration Choices Today 5 Data Integration Cost Considerations 6 Summary Executive Summary As long as there have been databases, there have been data integration challenges. With recent data formats such as text, objects and other multimedia forms, exponential growth of data, and global business expansion, data integration solutions have struggled to meet the demands of real-time decision making, operational effectiveness and cost-efficiency. This paper focuses on some fundamental choices you can make to help reduce cost while improving the value of your data integration solutions. One Database, One Application, One Solution? If the world were simple, you would only need to buy one application to manage all your customers, products, financials, suppliers and manufacturing. This one application would run on a single database that could grow to any size and still process data at lightning speed. For every question about a customer or a product, the answer would be at your fingertips, just ask your one solution. However, the world is not simple we need ways to achieve One View of the Truth of our businesses data. The way to make that happen is through data integration: technology and solutions to bridge the gap between disparate database information. 2

3 Data Integration Scenarios Data integration scenarios have countless variations. Three common situations are briefly described here and later used to identify cost-effective solutions to data integration: 1. Multiple, disparate data sources Example: An organization has invested in IBM systems and hardware. They have purchased various business applications (from IBM and others) for customer, product and financial management. Each of these applications (or suites) requires specific underlying databases. The organization is concerned with consistency of data and definitions, moving data quickly from one system to another, making sure data is error-free after movement, security/risk of private or restricted data, synchronizing data across multiple systems and system and business growth. 2. Similar data sources, large number of implementations Example: An organization has multiple global implementations of SAP applications running on Oracle databases, but data definitions, models and formats are different across each Oracle database instance. The organization is concerned with keeping applications and systems running without disassembling the existing data structure, being able to create master data scenarios and reducing the performance impact when sourcing data from various systems. If your business has specific data integration projects such as data migrations, replicating corporate data to backup systems or duplicating key data to low-cost databases for reporting and analysis, you should consider solutions that provide: Support for real-time (in addition to batch) data access and movement Support for any size data set Support for multiple databases (whether source or target databases) Support for any platforms If you need to support data synchronization activities, such as operational data sharing, real-time feeds to analytic systems or data balancing among numerous systems you should also require these key features: Support for data transformations, or at least open connectivity to include other data transformation/data quality solutions Support for data mirroring to data warehouse appliances The ability to read database transaction logs (to streamline database access) Change Data Capture technology to minimize access to the source and target databases Setting up a solid data integration solution that can address multiple databases, multiple platforms and real-time data access can solve many data management issues that arise from mergers/acquisitions, cloud computing requirements, hardware sunsetting, business changes (SaaS, etc.), compliance/risk, remote administration and data governance planning. 3. Non-integrated systems hardware and software all running independently Example: An organization has multiple data marts, data warehouses and application systems running separately, with executive reporting based on either manual or analyst-assisted data selection and management. There is no automation of data integration. The organization is concerned with the same issues as in (2), as well as making data available regardless of platform, how to prepare for data changes across different systems, providing a system to manage errors in data movement and preventing the high cost of losing data. 3

4 Data Integration Choices Today Today, data integration is available from a large number of sources, including hardware vendors, application suite vendors, database vendors open source software vendors and best-in breed data integration specialists. Vendor selection ultimately determines what features/functions are available for current projects as well as future projects. Investment in these vendor products may be short or long-term. Most data integration tasks are now handled by technology and a savvy group of IT or DBA staff familiar with data management techniques. Since data integration typically resides at the IT infrastructure layer at a company, IT staff members are usually the most familiar with the details of managing data traffic and manipulation. In some cases, business users or data stewards could be involved in data integration, depending on the complexity, interrelationships or end objective for the data. For the purpose of this paper, we are focusing on staff at the IT level and how to assist them in selecting successful data integration solutions based on cost and value. View some examples of data integration solutions for IT groups below: Multiple Data Sources, Multiple Implementations OVERALL COST COMPARISON 4 LARGE VENDOR STACK OPEN SOURCE SOLUTIONS BEST-IN-BREED DATA INTEGRATION Some software cost: some embedded data integration tools, others extra Software is free, but maintenance/ consulting are not $ Low software cost Limited support for competitive databases $ Limited support for various databases Incl Agnostic supports multiple databases and platforms $ Must build-in real-time capabilities $ Limited/ no support for real-time Incl Real-time (and batch) data access $ Limited support for log reading $ No support for log-reading Incl Supports log-reading Free upgrades $ No support for Change Data Capture Incl Supports Change Data Capture $ Consulting required Cost risk with open source code changes (revisions, staffing) Incl Quick to deploy, for IT users $ Long deployment $ Limited/ no error management Incl Built-in error management $ Long wait time/random customer support expertise $ Limited customer support $ Maintenance includes free upgrades Incl Immediate/dedicated customer support COST $ Large Vendor Stack Open Source Solutions $ Best-in-Breed Data Integration Similar Data Sources, Multiple Implementations Some software cost: some embedded data integration tools, others extra Software is free, but maintenance/ consulting are not $ Low software cost $ Limited support for complex integration flows $ Limited support for complex integration flows Incl Supports complex integration flows $ Must build-in real-time capabilities No support for log-reading, limited/ none for real-time Incl Real-time (and batch) data access $ Long deployment time Cost risk with open source code changes (revisions, staffing) Incl Supports Change Data Capture $ Consulting required $ No Change Data Capture Incl Quick to deploy, for IT users Free upgrades $ Limited/no error management $ Maintenance includes free upgrades $ Long wait time for customer support $ Limited customer support Incl Immediate/dedicated customer support COST $ Large Vendor Stack $ Open Source Solutions $ Best-in-Breed Data Integration Non-Integrated Systems Some software cost: some embedded data integration tools, others extra Limited support for competitive databases $ $ Must build-in real-time capabilities Software is free, but maintenance/ consulting are not Limited support for complex integration flows Cost risk with open source code changes (revisions, staffing) $ Low software cost Incl Incl Supports complex integration flows Real-time (and batch) data access $ Long deployment time $ No Change Data Capture Incl Supports Change Data Capture Limited/no support for data synchronization in different systems Limited/ no support for data synchronization in different systems Incl Supports data synchronization $ Consulting required $ No support for data recovery Incl Provides data recovery/backup Free upgrades $ No support for error-checking across different systems $ Maintenance includes free upgrades $ Long wait time/random expertise for customer support $ Limited customer support Incl Immediate/dedicated customer support COST $ Large Vendor Stack $ Open Source Solutions $ Best-in-Breed Data Integration $ Large Vendor Stack $ Open Source Solutions $ Best-in-Breed Data Integration

5 Data Integration Cost Considerations Large Vendor Stacks In many cases, large vendors have a number of products available in their stack, including hardware, software and middleware (data integration). They may also offer consulting services for their products. When buying a stack from one vendor, businesses can typically benefit from negotiated pricing (multiple purchases). However, be aware that data integration technology advertised as free in the package may not really be free. In many cases vendors will opt to minimize pricing on middleware and instead either increase/maintain high pricing on their other products, or maintain steep pricing on consulting, to offset the free middleware. Pricing distributed this way also reveals another data integration secret the vendor does not put as high a value on its data integration products as it does its other products (software suites or hardware.) This tells you development and support for those products do not warrant as high a regard either. When buying a stack consider that, in many cases, the software may be highly complicated or proprietary, resulting in high deployment/consulting costs, as well as require an ongoing commitment to pay consulting when anything goes wrong or changes need to be made in the customer s implementation. Costs to train are usually higher, and training may consume resource time and travel. Costs to replace staff may be higher with a more technical and complicated stack of products. Open Source Products On the opposite end of the spectrum is open source data integration software. While the open source data integration technology is available for free, the costs to deploy, maintain, and in some cases, run effectively, are not. Some costs for enterprise maintenance and support rival large vendor pricing. Most of the open source solutions are still largely untested other than in the most basic configurations, not in the enterprise scenarios that require support for disparate data types, complex data flows and multiple database versions. If your organization depends on data movement for its business, it might not be wise to depend on untested products that are subject to industry-wide technical evolution. As well, changes made in open code may or may not be good changes for your business and may require more investment in your inhouse staff or supporting systems to maintain. Best-in-Breed Data Integration Data integration specialists sit in the middle of the spectrum, offering moderate costs and flexible applications. Enterprise data is typically stored in some sort of database, but there are multiple variations of databases, the platforms on which they are deployed and where the platforms reside. Best-in-breed data integration specialists recognize this and have devoted engineering and technical/customer service to support many different databases and platforms. In some cases, data may appear through a channel, such as inbound XML, text or multimedia. To make use of this data within the requirements of the organization s business applications, it is commonly converted and deposited into a relational database. Best-in-breed data integration products are able to support multiple data formats, transform data between formats, provide functionality such as Change Data Capture to reduce time and stress on operational databases, and make real-time data access a reality. Data integration specialty software is an add-on cost to the data flow environment, but should be a relatively minor cost compared to the value of the combined database, application and platform investment. Even better are data integration solutions that can perform across multiple databases, rather than one data integration solution for each database. There is a significant cost savings in applying one solution to multiple uses. Look for data integration specialist products that support many databases and platforms. Data integration specialty software should also be easy to install, deploy and maintain. If you have to invest in DBA expertise, specific database expertise, expensive consulting projects and multiple servers/suites and other extraneous software just to get your data integration solution running, you will be spending a lot more than necessary to be effective in data integration! Data integration solutions should pay for themselves within one year (in some cases half that time) of implementation. Look for downloadable software, easy to understand guidance (wizards, good documentation) and a customer service support team that answers calls quickly with live support personnel. Each of these simple items can save thousands of dollars in staffing, go-live deadlines, support costs and your ability to deliver data as required by your business entities. 5

6 Summary When evaluating the data flow in your organization, consider data integration as an important function to enable real-time data access to all business entities. Reviewing key criteria in data integration scenarios relative to your business, as well as short and long term objectives, will help identify your data integration needs. Look for cost-effective, long-term solutions: solutions that will address as many functions as possible (including support for multiple databases and systems) at a cost relative to a percentage of your IT system investment. Look for solutions that will not require you to add more staff, consulting budget or time to your database projects all of which add cost. There are many data integration solutions available and your careful consideration of features, future development and associated costs will help decide the appropriate data integration solution for your business. HiT Software retains the sole proprietary rights to information contained in this document. No individual part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means without prior written permission from HiT Software, Inc. BackOffice Associates, LLC PO Box 808, South Harwich, MA T F info@boaweb.com Copyright 2013 BackOffice Associates, LLC. BackOffice Associates, Boring Go Live and all associated logos are trademarks or registered trademarks of BackOffice Associates, LLC in the United States of America and elsewhere. SAP as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and several other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates. All other products, company names, brand names, trademarks and logos are the property of their respective companies.