Master Data Management: Building a Foundation for Success

Transcription

1 WHITE PAPER Master Data Management: Building a Foundation for Success By Evan Levy, Baseline Consulting

2 This document contains Confidential, Proprietary and Trade Secret Information ( Confidential Information ) of Informatica Corporation and may not be copied, distributed, duplicated, or otherwise reproduced in any manner without the prior written consent of Informatica. While every attempt has been made to ensure that the information in this document is accurate and complete, some typographical errors or technical inaccuracies may exist. Informatica does not accept responsibility for any kind of loss resulting from the use of information contained in this document. The information contained in this document is subject to change without notice. The incorporation of the product attributes discussed in these materials into any release or upgrade of any Informatica software product as well as the timing of any such release or upgrade is at the sole discretion of Informatica. Protected by one or more of the following U.S. Patents: 6,032,158; 5,794,246; 6,014,670; 6,339,775; 6,044,374; 6,208,990; 6,208,990; 6,850,947; 6,895,471; or by the following pending U.S. Patents: 09/644,280; 10/966,046; 10/727,700. This edition published October 2009

3 White Paper Table of Contents Executive Summary... 2 Challenges with MDM Development... 3 The Five Core MDM Functions Data Cleansing and Correction Metadata Security and Access Services Data Migration Identity Resolution... 8 The Inventory for an MDM Foundation Data Cleansing and Correction Metadata Security and Access Services Data Migration Identity Resolution MDM Functional Requirements Conclusion About the Author Master Data Management: Building a Foundation for Success 1

4 Executive Summary A lot has been said and written about the challenges of implementing master data management (MDM). When it comes to determining the best solution for implementing subject area data integration, there are those who believe that purchasing an off the shelf MDM software product is the best and most cost-effective approach. Others think that developing a custom solution is optimal for deploying MDM in their organizations. Regardless of the approach, it s important that a company s MDM implementation doesn t occur in a vacuum. Because most organizations have already invested heavily in many of the technologies and methods commonly associated with MDM including data cleansing and metadata an MDM project needs to leverage any existing infrastructure capabilities to simplify integration into a company s existing IT environment. After all, implementing MDM without integrating it into the corporate information ecosystem simply defeats its purpose, which is to resolve the disparity of numerous data standards and integration methods within a single subject area. In our client work with MDM, we find that most of the goals aren t far reaching or revolutionary. In fact, MDM is best positioned and planned as a series of evolutionary processes in the improvement and linkage of key reference data. Such an approach needs to be well thought out and deliberate. Awareness of the phased nature of MDM has caused many IT managers to acknowledge that a so-called blank slate approach to MDM simply isn t practical. They realize that they have to implement their MDM programs in a manner that leverages a company s existing investment in technology and data management. MDM s promise is to deliver a single view of authoritative subject area data. In our book, Customer Data Integration: Reaching a Single Version of the Truth, Jill Dyché and I defined MDM as: The set of disciplines and methods to ensure the currency, meaning, and quality of a company s reference data that is shared across various systems and organizations. 1 The key word here is shared. As companies grow, reorganize, acquire other companies, install new systems, and share data within and outside of their firewalls, the need for common, authoritative reference data is expanding. When done right, MDM streamlines all of the rules and details involved in the deployment and maintenance of reference data for a single subject area. The purpose of this paper is to identify the challenges you may encounter with your MDM initiative and to discuss the key technology components that you will likely require. With that background, you will be more able to assess which components you already have at your disposal, which you will need to source from software vendors, and which you will build internally. 2 1 Jill Dyché and Evan Levy, Customer Data Integration: Reaching a Single Version of the Truth (John Wiley & Sons, 2006).

5 White Paper Challenges with MDM Development One of the biggest challenges in MDM implementation is developing a solid business case for MDM. But even companies that understand how to justify and pitch an MDM solution often fail to use the business case as the foundation for an MDM development framework. The business case should inform a set of business requirements that in turn inform a set of functional requirements. These functional requirements are the basis for a variety of MDM implementation decisions. Given the dramatic growth in MDM knowledge (as well as the evolution of associated technologies), there should be little surprise that implementation challenges have changed over the past five years. Although some people thought there were many parallels between MDM and the data warehouse, our experience with MDM adoption has shown a closer relationship between MDM and transactional development. The most visible challenges (or misunderstandings) that we see with MDM implementation include data model design, hub style selection, and subject area identification. One of the more prevalent misunderstandings is the relatively small scope that an MDM data modeling effort requires. Unlike an enterprise data model, the MDM data model need only reflect the identification attributes of a single subject area that are contained in the various systems of creation. For people or business-related topics, this is often less than a few dozen elements; MDM systems for inanimate subjects (e.g., part, product, and site) may require substantially more elements to support identification. The traditional complexities of developing and relating multiple subject areas within the data model aren t involved they are the domain of cross-functional databases. An MDM hub focuses on a single subject area. Where the data modeling differences become obvious is the need to capture and store MDM operational details (e.g., change logs, rules, security access, data correction). Most purpose-built MDM products already include data models to support these operational details. Although a hub can support multiple subject areas or domains, we have identified the best practice to be colocating multiple hub instances onto a single platform, not integrating the different subject areas into a single hub instance. Another common issue is selecting the hub style. There are two basic types of hubs: persistent and registry. The persistent hub stores all MDM-related details in one location; updates to any and all elements must occur on the hub. A registry hub only stores identification details for the customer or product (or other master data item) and links to the system of origin. All other descriptive details are retained within the system of origin. Other MDM styles are essentially hybrids of these two styles. Master Data Management: Building a Foundation for Success 3

6 In the early days of MDM, there was confusion about these two styles. However, our client experiences have proved that the registry style has become the more widely adopted of the two because it can support a more diverse set of technical requirements and typically requires less modification to the originating systems. We ve also seen a higher degree of development scalability with the registry style. We find that many of our clients don t fully understand their need for mastering the different subject areas (or data domains) within their enterprise. This misunderstanding often occurs because of the funding process of most IT organizations; the MDM initiative exists because of the business challenges associated with a few applications struggling to share and synchronize common master data. MDM is often positioned as a development enhancement, not an infrastructure investment. Consequently, the MDM initiative is focused on and limited to only a few application systems and one subject area. To ensure that your MDM investment reflects your company s needs and priorities, you should undertake a requirements analysis that reviews the individual reference data needs of multiple operational and analytical applications. This analysis will not only yield the specific requirements and rules on behalf of an individual subject area but also identify the relative priority and value of the different subject areas that need to be mastered. This level of detailed analysis can serve as an insurance policy against over- or under-investment in your hub development project. Once we help our clients deliver business and functional requirements for MDM, we typically assess the current IT infrastructure to determine which key technical or processing capabilities already exist. Frequently, we find that many of our clients have already invested in the technical areas associated with MDM (e.g., data quality, data management, and security). In fact, many have done a lot of the heavy lifting when it comes to establishing standard methods and practices with their existing infrastructure tools and capabilities. Managers at these companies often admonish us to support existing capabilities and invent only what s necessary. The goal of an MDM project should be to deploy reconciled master data quickly and efficiently. There s no point in replacing technology and methods that already work. The Five Core MDM Functions Depending on the maturity and size of an IT organization, there may be a sizable collection of infrastructure services associated with managing data and systems. In our experience working with firms on new MDM programs, there are typically five technical functions that are core to MDM processing. Frequently, these capabilities are already part of the IT infrastructure. The five functions are: 1. Data cleansing and correction 2. Metadata 3. Security and access services 4. Data migration 5. Identity resolution Ironically, companies usually have acquired these technologies for non-mdm processing needs. Nevertheless, they can be brought to bear on MDM-specific functions. The following sections review each of these capabilities as well as highlight circumstances where you may have existing tools or assets that can be used or repurposed. We ll also identify a checklist of items to review or investigate within your own environment to help you prepare for MDM, regardless of whether you build or buy your MDM solution. 4

7 White Paper 1. Data Cleansing and Correction Data cleansing is fairly common within most IT organizations, particularly when data is highly prone to data entry errors. Most data quality environments have been set up to focus on the cleansing and correction of well-understood data values. Customer name and address values are frequently the starting point for many companies seeking to clean up their bad data. Commercial data quality tools have built-in logic and rules to support the interrogation of these addresses to generate an accurate value. They simply compare an input value to their own address database and determine (through various means) if the address is accurate or if it needs to be corrected to reflect an accurate address. Most data cleansing tools can be configured with rules to support custom data-types, too (e.g., ID numbers or colors).the key to the successful use of data cleansing technology is the creation and management of the data cleansing rules contained within the tool. As organizations mature with their use of data cleansing tools, they ve typically built a large number of rules to support the various data-types used across their technology environment. Cleansing Identification 123 Main Ave, Winter Park, FL 123 Main St, Orlando, FL Data Cleansing Christopher Lee 123 Main Ave Winter Park, FL Match Processing Security CRUD Processing MDM System Figure 1. Data cleansing process within a MDM System Although some people may view data cleansing and correction as a commodity, it s a highly sophisticated process that isn t trivial for a programmer to recreate, as seen in figure 1 above. (This is why companies purchase data quality software.) Because part of implementing MDM is ensuring that the individual reference data elements are clean and correct, it only makes sense for any implementation approach to leverage existing data cleansing methods already in place within an IT organization. In addition, because MDM is typically implemented as a centralized processing system, it is a natural platform for delivering cleansed reference data to the numerous systems that require it. The challenge many companies have in deploying data quality is that often every business application defines its own custom set of cleansing routines. Because MDM is focused on ensuring a consistent set of reference details, any effort to compare or correct data values occurs within a single set of automated processes. To ensure a single, central reference list, there should be a single, sanctioned solution for cleansing reference data. The benefits become clear: establishing and using a single set of cleansing rules, support for centralized (and lower cost) processing, and the ability to ensure consistent cleansing across all systems. Master Data Management: Building a Foundation for Success 5

8 2. Metadata Many companies we work with use metadata to address the disparity of data across numerous heterogeneous systems. As companies have grown, the number of systems they require to run their operations has also grown. Without metadata in place, there is no practical way to understand or keep track of the data that exists across a company s various systems. One of our retail clients has to reference an individual product in three different ways because each of its main operational systems (POS, distribution, and replenishment) uses a different product identifier to address operational processing. Metadata isn t limited to identifying the individual data element names and their definitions. It also identifies the data values and the origin of that data (its lineage). Since many companies have already implemented technologies such as data warehouses, an enterprise service bus, and ETL tools, they likely already have a metadata solution in place to identify the data within the individual systems as well as to track lineage as the data moves between the different systems. But metadata isn t MDM. Metadata focuses on the descriptive detail about data. It doesn t resolve duplicates. It doesn t determine data s accuracy. And it doesn t identify synonyms or link different value representations of the same item it doesn t know that UPC is the same as SKU , for instance. Metadata only provides descriptive attribute information about the data itself; it doesn t reconcile, fix, or link that data to a larger set of systems or processes. If you re going to establish a single view of reference data that all systems have access to and will use in other words, if you re going to build an MDM solution those systems must understand how to reference the individual details of those data elements and understand their attributes. An MDM solution can use existing metadata by leveraging existing data definitions and standards. The most visible metadata features that are useful in an MDM environment include: Terminology or data element names for instance, the data element name is ItemColor Data values for instance, the acceptable values are red, green, or blue Value representation for instance, CC0000, 00CC00, and 0066CC Lineage details for instance, Created: 01/12/2009. System of origin: Order System Data definitions for instance, the color of the item The use of existing metadata structures can dramatically simplify other systems abilities to adopt and use MDM because they may already recognize and work with the existing metadata. Integrating with an existing metadata environment can mean that the MDM development team won t need to reconcile differing views of the different application systems because the existing metadata system has already addressed that detail. 3. Security and Access Services One of the lesser-known functions within MDM is the need to manage access to individual reference data elements. MDM systems typically manipulate data using CRUD processing to create, read, update, and delete so centralizing the permissions to access and change key data is an important part of maintaining master data. MDM can support a very detailed or granular level of security access to the individual reference data elements. One of our clients, a consumer packaged goods firm, differentiates MDM access because only a limited number of individuals are allowed to create or delete items within the company s product catalog; a larger group of individuals can update specific aspects of those products. MDM allows security and access to focus on specific operations in specific reference item attributes. 6

9 White Paper Companies have invested heavily in security technologies already, be it single sign on, LDAP functionality, or one of many other access control systems. These capabilities are important in their own right because they allow IT departments to manage access across multiple systems and track access to often-sensitive data. In the world of distributed processing where users can access information via a desktop, a laptop, or even a handheld device, companies rely upon centralized security services to manage, track, and monitor data access. Because of the granular level of data access that MDM affords CRUD processing against individual reference values an MDM system should avoid having its own siloed proprietary security solution. Instead it needs to interface to existing security services that an IT department relies on to manage application access. Although many purpose-built MDM products already provide interfaces to support interacting with an environment s existing security services, any homegrown MDM solution needs to address data access security in a manner that can leverage existing policies and access technologies. 4. Data Migration Data migration technologies alleviate the need to develop specialized code to extract and transfer data between different systems. Companies have invested heavily in ETL tools and application messaging technologies such as enterprise service bus (ESB) because of the need to move large volumes of data between their various application systems. The challenge in moving data between systems exists because the data is rarely stored in a simple, intuitive structure. In most instances, an application s data is stored within a proprietary structure or database to support its own unique processing requirements. Access and navigation to a specific set of data elements for non-application purposes can be complex and cumbersome. Many ETL tools make it simple for developers to move data between systems without having detailed knowledge of the individual systems; the tools understand how to interact with many different types of databases and packaged applications. The same holds true for the application messaging or ESB technologies. Many companies have standardized on ESBs for moving transactional data between application systems quickly and reliably. Most of these product vendors have developed prebuilt modules and interfaces to the most popular databases and packaged software products to simplify development efforts in migrating data between disparate systems. Regardless of the specific tools and technologies used in transporting data into or out of an MDM system, there are two basic ways of moving data: large volume bulk data (loads or extracts) and individual transactions. To load the MDM server initially with data from an application system requires bulk data loading capabilities. ETL tools are very well equipped to handle the application data extraction and MDM system loading. Likewise, most purpose-built MDM products have been designed to support connecting to a company s various application systems using an ESB (or similar technology). This approach ensures that data is propagated to/from the applications and MDM systems at transaction-level speeds. Regardless of whether you buy or build your MDM solution, the MDM system cannot sustain its own custom data migration network and should instead rely on the incumbent data migration environment. Master Data Management: Building a Foundation for Success 7

10 5. Identity Resolution We work with a large, multinational financial services firm on Wall Street. At this company, individual financial products are managed and supported on separate operational systems. To have a customer-centric view of the business that is, understand the various individual products associated with each customer the bank needed to determine a means of relating the individual account holders across the individual products. It purchased identity resolution software to analyze and link the individual clients across each of the systems. This technology allowed the bank to quickly determine whether multiple instances of a single individual were the same person. It found this tool to be valuable in linking external information to the individual account holders, too (credit, household, and legal details). The bank later applied this technology in other areas of the company to address identify theft, alias identification, and other fraud situations. Most MDM solutions also benefit from the ability to uniquely identify and manage individual reference details. In the situations of simple subject areas (like color or size), managing and tracking the various reference values is very straightforward. This simplicity comes from the fact that the values are easy to differentiate; there s not much logic involved in determining if the value matches red, green, or blue. Complexity occurs when the subject details being mastered have more variables, and variables that can be vague such as a person. In the instances of complex subject areas, mastering the reference value requires more sophisticated analysis of the numerous attributes associated with the individual reference value such as their name or address. An example of this is determining if two systems containing records for Chris Lee are referring to same person, or two different people, perhaps Christine and Christopher Lee.? Christine Lee 123 Main St Orlando, FL Christopher Lee 123 Main Ave Winter Park, FL To determine if these two records refer to the same individual, the MDM system will have to analyze multiple data elements associated with the person. To do so requires leveraging the data and infrastructure capabilities that already exist within most IT organizations. To establish if these values match, a series of functions can be applied to the data. To support comparing disparate data values, you need to cleanse, correct, and standardize the data before processing (or comparing) values. 8

11 White Paper The Inventory for an MDM Foundation In working with clients to determine how to best launch an MDM effort, we frequently approach initial requirements and architecture efforts by asking, What do we need, and what do we have now? Here are some techniques for you to use to assess how to move forward on your own MDM development effort. 1. Data Cleansing and Correction Identify data cleansing tools that are already implemented within your company on other projects; pay special attention to applications that are likely to link to the MDM system Determine the data cleansing rules for the master data subject area and whether they have been documented Establish how you will interface your MDM system to the data cleansing tool (e.g., Web services or APIs) Contact the product vendor to determine which MDM product(s) it may already support 2. Metadata Review existing data warehouse metadata to discover whether there is metadata content that may apply to the master data subject area in question Determine whether there are metadata standards in place that apply to the given master data Find out if any of the key application systems use existing metadata standards If your company has a data management team, see if there is already a sanctioned process for identifying and developing new metadata content 3. Security and Access Services Identify the level of security requirements that need to apply to core master data Talk to the application architecture group to determine if there are application-level security standards in place Investigate whether your company has already defined a set of security and access Web services that may be leveraged for MDM 4. Data Migration Identify the bulk data migration standards that are in place for example, how is Informatica technology used in implementations other than data warehouse or BI implementations? Determine the current mechanism for application-to-application connectivity: Is it EAI? ESB? Point to point? Clarify which application systems can accept reconciled and corrected data Determine which legacy systems can only support bulk data extract versus transaction access Master Data Management: Building a Foundation for Success 9

12 5. Identity Resolution Understand if your company already has identity resolution software in place to identify the existence of multiple customers, products, or other subject area items; this type of capability is most likely to exist in a customer-facing financial system Investigate whether the existing identity resolution system is configurable with new or custom rules Determine how the identity resolution technology interfaces to other systems (e.g., embedded code? API? Web services?) 6. MDM Functional Requirements Identify a handful of high-profile systems that depend on the timely sharing and synchronization of reference data; if you have a BI environment, we recommend looking at the predominant source systems as well as the most-used data marts Decide which reference data is shared and synchronized between multiple systems; the need for operational integration typically reflects higher business value Determine the individual system s functional requirements for its reference data (e.g., create/ read/update or data quality/correction) Categorize the individual data elements by subject area to identify the specific subject areas requiring mastering. It s only practical to implement MDM one subject area at a time. Conclusion When it comes to the question of build versus buy, for MDM the answer is as complex as your company s IT infrastructure, functional requirements, and business needs. The considerations are multifaceted and require an intimate knowledge of what you can do today versus what you ll need to accomplish tomorrow. When it comes to integrating master data, one thing is clear: The first step to ROI is avoiding unnecessary expenses. Investigate and use the capabilities that already exist within your IT infrastructure, enabling you to make an informed decision on launching MDM the right way. 10

13 White Paper About the Author Evan Levy is a partner at Baseline Consulting ( a consulting firm specializing in business analytics, data warehousing, data governance, and data integration. In addition to his management responsibilities at Baseline, Evan is actively involved in guiding client projects. Considered an industry leader on the topic of data integration and data management, Evan is a faculty member of The Data Warehousing Institute and is a speaker and contributor to TechTarget, DMRadio, and the MDM Summit. His blog ( focuses on data integration and the associated architectural and political challenges within IT organizations. He is co-author of the book, Customer Data Integration: Reaching a Single Version of the Truth (John Wiley and Sons, 2006). Master Data Management: Building a Foundation for Success 11

14 12

15 White Paper Master Data Management: Building a Foundation for Success 13

16 Worldwide Headquarters, 100 Cardinal Way, Redwood City, CA 94063, USA phone: fax: toll-free in the US: Informatica Corporation. All rights reserved. Printed in the U.S.A. Informatica, the Informatica logo, and The Data Integration Company are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners. First Published: XXXX 7069 (11/03/2009)