The Growing Practice of Operational Data Integration. Philip Russom Senior Manager, TDWI Research April 14, 2010

Transcription

1 The Growing Practice of Operational Data Integration Philip Russom Senior Manager, TDWI Research April 14, 2010

2 Sponsor: 2

3 Speakers: Philip Russom Senior Manager, TDWI Research Gavin Day VP of Operations and Information Technology, DataFlux 3

4 Agenda Defining Operational Data Integration (OpDI) Three Practices of OpDI Growth of OpDI OpDI Project Types Organizational Issues for OpDI Software Automation for OpDI Recommendations 4

5 Definitions of Operational Data Integration OpDI is manifested in diverse project types, so it s hard to define OpDI with one, simple description. Sometimes defined negatively from BI perspective as: non-data-warehouse data integration work POINT: OpDI rarely has much connection to BI/DW More positive definitions based on what OpDI does: the exchange of data among operational applications, whether in one enterprise or across multiple ones Applies to some forms of OpDI, like data synchronization and business-to-business data exchange data and database maintenance work that relocates datasets or database management systems Applies to OpDI forms, like data migrations, consolidations, collocations, and database mgt system upgrades. 5

6 Three Broad Practice Areas of Data Integration Business Initiatives and Technology Implementations Data Integration Practices Data Warehousing (DW) Business Intelligence (BI) Analytic Data Integration (AnDI) Customer Data Integration (CDI) Master Data Mgt (MDM) Product Information Mgt (PIM) Hybrid Data Integration (HyDI) Data Migration Data Synchronization Business-to-Business (B2B) Data Exchange Operational Data Integration (OpDI) Focus of today s Webinar 6

7 The Three Main Practice Areas within OpDI 1. Data Migration Four types of database maintenance projects: Database migrations, consolidations, and collocations, plus database management system upgrades 2. Data Synchronization (or data sync) Usually about syncing customer data across multiple customerfacing apps, e.g., CRM, SFA, call center, etc. Also enables active-active and multi-master database configurations for database high availability 3. Business-to-Business (B2B) Data Exchange Exchanging documents and files across diverse organizations involved with supply chains, monetary exchanges, procurement, fulfillment, BPO outsourcing, etc. 7

8 OpDI is a Fast- Growing Practice Over time, OpDI has grown as a percentage of overall DI work, relative to AnDI: 81% AnDI and 19% OpDI in % AnDI and 49% OpDI in 2008 Rounded: 80/20 to 50/50 split OpDI growth is a dramatic change that only took four years. OpDI needs more people and budget to keep pace with growth. Both AnDI and OpDI are growing. OpDI is growing faster than AnDI. Both practices need resources. DI work is getting more varied and projects are more numerous. DI projects for both BI/DW & op.apps are getting bigger. With data integration usage in your organization, what is the approximate percentage split between analytic DI versus operational DI? November 2004 (Source: Forrester Wave Report) February 2006 (Source: TDWI Tech Survey) February 2007 (Source: TDWI Tech Survey) November 2007 (Source: TDWI Tech Survey) August 2008 (Source: TDWI Tech Survey) December 2008 (Source: TDWI Report Survey) Analy tic Data Integration (AnDI) 61% 63% 51% 81% 75% 75% Operational Data Integration (OpDI) 39% 37% 49% 19% 25% 25% 8

9 Drivers for OpDI Growth, per Practice Data Migration As number of databases increases, maintenance does, too Databases proliferate and age, so need migration/consolidation Mergers/acquisitions & reorganizations force migrations Data Synchronization As customer-facing apps increase, data sync does, too 360-degree views are becoming the norm Business-to-Business (B2B) Data Exchange For firms with active supply chain, B2B data exchange is critical As business goes more global, as global firms integrate regional business units more, B2B data exchange increases (to reach internal units, plus external partners & customers) Many old solutions; firms are forced to update/replace them 9

10 Why care about OpDI Now? OpDI is a growing practice. It s an increasing percent of DI workload. OpDI solutions need improvement or replacement. Many are hand-coded legacies; replace them with vendor tools. OpDI solutions tend to be feature poor. Augment them with missing functions for data quality, master data, scalability, architecture, Web services, development tools. OpDI and AnDI have different goals and sponsors. Don t assume you can do both with same team, tools, budget. OpDI addresses real-world problems, especially with change. Data migration is essential to enterprise transformation. OpDI supports mission-critical applications. Enterprise comes to a halt without data sync & B2B data exchange. 10

11 Data Migration Project Types* Database Migration From database platform A to platform B. It s more like new dev than migration. Database Mgt System Upgrade Version A of Brand X upgrades to version B of Brand X. Database Consolidation Multiple, similar databases consolidate into a single one, with a single data model. Database Collocation Multiple databases are copies into a single DBMS instance (typically on a single hardware server), without merging data models. One-to-One Brand A: Old and/or non- Standard Many-to-One Redundant Databases One Data Model Many-to-One Redundant Databases Brand B: New and/or Standard One-to-One Version A of Brand X Version B of Brand X Multiple Data Models * Synonyms = implementations, initiatives, jobs, tasks, assignments 11

12 Data Migration Projects These projects are in the context of: General database maintenance or data lifecycle mgt In support of applications maintenance E.g., when application instances are consolidate or apps are migrated to a new platform, application databases are, too. Reduce the number of databases to reduce cost Migrate data to more modern and/or standard platforms Data migration projects kill off systems Far more intrusive than data sync Can be a multi-phase project that runs for years Coordination among data & app owners, sponsors, business managers, end-users, multiple IT teams, etc. Multiple projects may be combined: E.g., before consolidating several databases, migrate some of them to common platform & upgrade others to a desirable version Collocate several databases before consolidating them into one; typical of data mart consolidation projects 12

13 Data Synchronization as a form of Operational Data Integration Many-to-Many Most common data sync solutions: Sync customer data across customer-facing apps Sync operational data across ERP instances and other apps Sync primary and secondary databases for high availability Data Synchronization and Replication overlap somewhat The two use similar technologies and get similar results Most data sync solutions are built with replication technology Other tech s can also implement or contribute to data sync Including EAI, ETL, and hand coding One-to-One Data sync solutions involve advanced capabilities rarely seen in simple replication solutions. (See next slide.) 13

14 Advanced Capabilities of Data Synchronization Multi-directional data flow. Usually bidirectional, not just one-way May also be multi-directional Conflict resolution. Data value conflicts arise when syncing Supports rules for resolving conflicts Heterogeneous sources and targets. Handles many database brands, app & file types Supports appropriate interfaces for all these Data transformation. Normalizing, merging data from numerous heterogeneous data models 14

15 B2B Data Exchange As a form of Operational Data Integration What it does: Documents or files containing data are exchanged between two or more organizations Examples: flat files, XML files, EDI documents, spreadsheets Relationship of organizations exchanging data varies: Exchange data between business units of the same company Among companies in a B2B relationship Most B2B relationships involve exchanging hard goods: Manufacturing, retail, consumer packaged goods, etc. Other relationships involve exchanging information or money: Financial services, insurance, healthcare, government, etc. B2B Data Exchange enables communication among organizations that cannot communicate data directly: Two organizations have very different data models So data exchange usually involves third, intermediary model 15

16 B2B OpDI Hinges on Intermediary Data Models 1 DI routines transform data from enterprise models into the appropriate exchange model. 2 Know and support the lingua franca that all organizations can speak. Transport datasets with security and scalability. 3 DI routines transform data from exchange models into the appropriate enterprise models. Business A B2B Data Exchange Business B Applications Operational Transactional Miscellaneous Data Sources Data Integration Get enterprise data Transform it into exchange format Data Quality Standardize Verify Intermediate Data Models XML-based open standard Other standards Proprietary data model to which partners agree Data Integration Get exchange data Transform it into enterprise models Data Quality Verify, re-standardize Process exceptions Applications Operational Transactional Data Warehouse 16

17 DataFlux: OpDI in Action

18 The Tools to Succeed Foundation in data governance Ability to create, enforce and monitor business rules across your organization Visibility and transparency to ensure your data reflects your business Business and IT collaboration Business builds the rules; IT manages and applies the rules IT and business goals and process alignment Build once, use repeatedly Utilize web services and other industry standards to simplify the integration and synchronization process Deliver a foundation for repeatable data integration programs (and realize rapid time-to-value)

19 Data Migration A US-based manufacturer of document management products needed a more unified view of its expanding European operations Goal Create a single instance of an enterprise resource planning (ERP) application to better manage operations worldwide. Challenge Solution Results The company s European group had grown through a mixture of organic growth and M&A. As a result, the operations group was using 20+ ERP systems across the European region each localized to a different country. Performed a pilot data integration project to create a single set of business rules for migrating data, then expanded that to bring other geographies online. The European operations finally had a way to look across the continent to make decisions. Logistics were improved due to shared resources, and the company was able to capitalize on new markets more rapidly. 19

20 Data Synchronization Leading provider of automotive information seeks to provide more accurate data on transactions from millions of consumers. Goal Build a data quality and data integration backbone that validates data as it enters operational systems. Challenge Solution Results With data coming from a variety of internal and external data sources, the company struggled with: Inconsistent data that was not standardized across systems Millions of records that could not easily be merged or consolidated effectively Used real-time data integration technologies (with web services on an EAI backbone) to validate data entered into systems prior to moving it to a system of record. Data is validated as it enters the target systems providing consistent, accurate and integrated data in a timely fashion.

21 Data Exchange Clinical research organization (CRO) with hundreds of clinical trials underway at any time across the globe. Goal Understand information on patients, doctors and projects across systems. Challenge Solution Results Data on clinical trials was kept in different applications and databases. This organization wanted to make use of that data, but each database had its own standards and nomenclatures, making integration difficult. Used a combination of batch and real-time data integration technologies to deliver a unified master data hub. The project started with data profiling, moved to data quality/ governance, and finished with master data management that synchronizes enterprise data. Delivered a more unified view of trials, which helps the organization make smarter decisions about future projects.

22 Data Management Methodology 22

23 Staffing OpDI For DI outside data warehousing, who does the work most often? How many full-time people were assigned to your most recent OpDI project? Source: TDWI Report Survey of late 2008, 336 respondents. Source: TDWI Report Survey of late 2008, 252 respondents. Which of the following best describes the frequency of OpDI work? 23

24 Tools and Techniques for OpDI In your organization, what is the preferred technology for most operational DI projects? In addition to the preferred technology you selected in the last question, which of the following techniques and tool types were used? Source: TDWI Report Survey of late 2008, 252 respondents. 24

25 Recommendations Expect both OpDI and AnDI to grow; OpDI needs more resources. Revamp your OpDI solutions for future success. Many are hand-coded legacies that need replacing Others lack modern functions for quality, master data, services, cloud Apply multiple OpDI project types in tandem. Collocate or upgrade databases before migrating/consolidating them. Sync old and new databases during a migration/consolidation. Foster data standards; these are key to B2B data exchange. Avoid staffing OpDI from the data warehouse team. They have great skills, but you delay important BI work. Consider a data integration competency center. It provides staff, shared services, other resources for all DI practices. When OpDI work is intermittent: Outsource it. Instead of permanent staff, hire consultants Look for tools licensed via short-term contract, open source, SaaS, or Cloud 25

26 One last recommendation Get a free copy of TDWI Report on Operational Data Integration Download the report in a PDF file at: reportseries It goes into more detail than today s Webinar, so read it. 26

27 Questions?? 27

28 Contact Information If you have further questions or comments: Philip Russom, TDWI Gavin Day, DataFlux 28