WHITE PAPER. How to Launch a Data Governance Program: Practical Guidelines for Technology Implementers

Transcription

1 WHITE PAPER How to Launch a Data Governance Program: Practical Guidelines for Technology Implementers

2 This document contains Confi dential, Proprietary and Trade Secret Information ( Confi dential Information ) of Informatica Corporation and may not be copied, distributed, duplicated, or otherwise reproduced in any manner without the prior written consent of Informatica. While every attempt has been made to ensure that the information in this document is accurate and complete, some typographical errors or technical inaccuracies may exist. Informatica does not accept responsibility for any kind of loss resulting from the use of information contained in this document. The information contained in this document is subject to change without notice. The incorporation of the product attributes discussed in these materials into any release or upgrade of any Informatica software product as well as the timing of any such release or upgrade is at the sole discretion of Informatica. Protected by one or more of the following U.S. Patents: 6,032,158; 5,794,246; 6,014,670; 6,339,775; 6,044,374; 6,208,990; 6,208,990; 6,850,947; 6,895,471; or by the following pending U.S. Patents: 09/644,280; 10/966,046; 10/727,700. This edition published June 2006

3 White Paper Table of Contents Executive Summary Governance by Spreadsheet: The Road to Perdition Effective Data Governance: It Takes a Platform Six Criteria for a Data Integration Platform Architecture: SOA and Data Governance Organization: ICCs and Data Governance How To Get Started: Four Things Your IT Organization Can Do Now Tie the Pilot to a Key Business Goal Make Pilots a Spreadsheet-free Zone Talk to the Enterprise Architects Get IT Plugged into the Data Governance Structure Conclusion and Next Steps About Informatica How to Launch a Data Governance Program 1

4 DEFINITION OF DATA GOVERNANCE The processes, policies, standards, organization, and technologies required to manage and ensure the availability, accessibility, quality, consistency, auditability, and security of data in a company or institution. Executive Summary More and more companies and government organizations are recognizing the value of data as a strategic asset an asset to be carefully developed and managed. However, many organizations have not yet evolved their processes, policies, and infrastructure to be able to cultivate their data as an asset. As a result, data governance is an emerging discipline which many organizations are beginning to adopt. Data governance is defi ned as the processes, policies, standards, organization, and technologies required to manage and ensure the availability, accessibility, quality, consistency, auditability, and security of data in a company or institution. Research shows that the majority of organizations that are implementing data governance programs are in the planning or early pilot phase. In 2005, The Data Warehousing Institute (TDWI) surveyed 750 respondents to ask them about the status of their organization s data governance initiative. Forty-two percent of the surveyed reported that their organization had no plans for a data governance initiative, while 33 percent reported that plans for a data governance initiative were under consideration. Seventeen percent reported that their data governance initiative was in the design or implementation phase. Only 8 percent had deployed a data governance initiative. 1 Why have many organizations hesitated to implement data governance programs? Common challenges of early phase programs include: Diffi culty in obtaining executive support Poor organizational alignment Challenges developing the business case While these business and organizational issues need to be resolved if a data governance program is going to succeed, another common problem demands attention: the lack of a robust technology infrastructure specifi cally a data integration infrastructure to support data governance standards and processes. This white paper focuses on this technology infrastructure issue. After reading this white paper, you ll understand: Current data governance approaches and their limitations Key technical requirements for a data integration infrastructure Best practice architectural approaches The role of competency centers in managing the data integration infrastructure How to get started with data governance programs, including practical guidelines and a checklist for evaluating the technology infrastructure to support such a program 2 1 Russom, Philip. Taking Data Quality to the Enterprise through Data Governance, TDWI, March Available online at

5 White Paper Governance by Spreadsheet: The Road to Perdition When launching a data governance program, many organizations start by defi ning the program s organizational roles and responsibilities and establishing governance processes. They also launch pilot efforts to assess existing data, defi ne its business rules, rationalize it, and improve its quality. More often than not, however, organizations undertake these pilot efforts in an ad hoc manner. Most problematically, many organizations attempt to implement data governance programs by SQL and Excel that is, by running manual queries to profi le existing data and capturing data defi nitions and rules in spreadsheets. These manual tools require a lot of highly repetitive effort and yield low productivity. And they lack mechanisms for documentation, process, or policy enforcement. Some organizations are using a collection of more automated tools to address their data requirements. Data analysts may use ERwin or Rochade to model the data. FirstLogic or Trillium may be used to cleanse customer name and address data. An ETL (extract, transform, load) tool may be used to transform disparate data from multiple systems. However, while automated, this approach relies on the use of multiple, disparate tools, each of which addresses a narrow scope of functionality. When organizations launch data governance programs, they often adopt these siloed tools and approaches out of convenience. Staff are already familiar with the tools; therefore, it is easy to get a pilot project started. No training is required, and organizations can avoid the political opposition that sometimes arises when new tools are imposed. Yet the limitations of a siloed, tool-based approach can prevent a data governance program from achieving its goals and scaling effectively. The most obvious problems are lack of automation, resulting in repetitive manual labor, and the costs of maintaining redundant tools and skillsets. However, there are several more signifi cant consequences which negatively impact the data and the data governance processes themselves, including: Proliferation of data connectivity and access points. Data governance should rationalize and control access points to data. But when different groups use different tools to access and integrate data, data connectivity is fragmented. This can lead to the proliferation of data silos, as well as diffi culty accessing data, particularly data in arcane legacy systems. Challenges delivering data as needed. For data to be useful, it must be delivered downstream to applications and users when needed and in the format required. To drive business operations, that means the data must be delivered in a highly performant, highly available manner. With ad hoc approaches and tools, it is nearly impossible to guarantee the level of reliability and scalability required for mission-critical operations. Diffi culty resolving data quality issues in a systematic manner. It s one thing to identify and resolve data quality issues within a specifi c data set at a point in time. Implementing an ongoing data quality improvement and monitoring program, addressing all types of data (e.g., customer, product, fi nancial, etc.) across the entire data lifecycle, is another matter entirely. A holistic data quality program cannot be implemented using narrow tools focused specifi cally on profi ling or on name and address cleansing. How to Launch a Data Governance Program 3

6 Lack of consistency in data defi nitions, rules, and processes across different tool sets, application domains, and business units. One of the primary goals of data governance is to ensure the consistency of data across the enterprise. When different approaches are used to integrate and manage data, inconsistencies in the data itself will inevitably arise, in large part due to the lack of integrated metadata. This is particularly problematic in the area of compliance reporting, where it is absolutely critical to provide a single version of the truth to regulators. Poor documentation of the data lifecycle. In today s increasingly strict regulatory and corporate governance environment, organizations need to properly document all data, preferably by capturing logic and processes as metadata, to certify its validity. The inability to trace the use of data across multiple systems, particularly with the heavy use of spreadsheets for documenting data and data rules, leaves organizations exposed to potential compliance and governance problems. Potential security exposures. Business data must be secure, particularly sensitive fi nancial and customer information. Using a collection of different tools makes it extremely diffi cult to enforce a consistent and comprehensive policy for securing data, including establishing rules for who can access or change it. Effective Data Governance: It Takes a Platform Of course, data governance is about much more than just technology it s about standards, policies, and processes, and organization. But without the right data integration technology infrastructure, it s nearly impossible to ensure that data is accessible, available, of high quality, consistent, auditable, and secure across the enterprise. Figure 1 shows the building blocks of an effective data governance program. Automating data governance processes, enabling organizational collaboration, and systematically enforcing data standards and policies demands the right technology infrastructure. To drive robust, scalable data governance programs, organizations are now implementing a common, enterprise data integration infrastructure one that leverages a unifi ed data integration platform, is built on shared services, and supports competency centers. IT organizations charged with implementing data governance programs must ensure that they select a data integration platform that has the required functionality, while also supporting the architectural and organizational principles required for data governance. Data Governance Data Accessibility Data Availability Data Quality Data Consistency Data Security Data Auditability Data definitions & taxonomies Standards Master/reference data Policies & Processes Data definition Monitoring & measurement Roles & responsibilities Organization Training & education Enterprise data model Technology & tools standards Data access & delivery Data change management Planning & prioritization Org. change management Data Integration Infrastructure Figure 1: The Building Blocks of an Effective Data Governance Program 4

7 White Paper Six Criteria for a Data Integration Platform IT organizations can use this checklist of evaluation criteria to ensure that the data integration platform they select offers the comprehensive set of capabilities required for a robust data governance program. It s critical that a single, unifi ed platform supplies these capabilities to ensure consistency and reuse and to provide uniform process and policy controls. 1. Data Accessibility The platform should ensure that all enterprise data can be accessed, regardless of its source or structure. Supported data types. Is the platform able to access the following data types with one tool and leveraging common metadata: mainframe data; structured data; unstructured data (e.g., Microsoft Word documents and Excel spreadsheets); XML and EDI data; relational data; application data; and message queue data? Pre-built connectivity. Does the platform have pre-built connectivity to a wide variety of systems, including multiple mainframe formats, messaging systems, and numerous applications? Input/output data validation. Does the platform validate input/output data? Event logging. Does the platform provide failed session statistics, error messages, metadata statistics and lineage that help assess the exceptions and failures related to accessing data? Federated access. Does the platform provide both physical and virtual/federated access to data in one common tool? Cross-fi rewall access. Does the platform support secure, highly performant data movement across fi rewalls? 2. Data Availability The platform should ensure that data is available to users and applications when, where, and how needed. Throughput. Does the platform make it easy to confi gure multiple performance enhancement options including pipelining, dynamic partitioning, and smart parallelism? Scalability. Does the platform take advantage of 64-bit, thread-based parallel processing and grid deployment for near-linear scalability? Automatic failover and recovery. Does the platform feature automatic failover and recovery capabilities? Does it provide a graphical status on the grid, as well as other key indicators/ alerts? High availability. Does the platform enable you to easily confi gure high availability? Does it include built-in resiliency, failover and recovery? Does it support multi-node/grid deployment? Volume and timing. Does the platform allow data volumes and latencies (e.g., large volume batch vs. message-based real-time) to be confi gured to meet business needs, without any recoding? Breadth of delivery protocols. Can the platform be easily confi gured to deliver data via different protocols and methods, including loading physical databases for SQL-based access, creating virtual data views (EII), publishing to a message bus or queue, and publishing Web Services? How to Launch a Data Governance Program 5

8 3. Data Quality The platform should ensure the accuracy and validity of data. Profi ling. Does the platform include tools to automatically profi le data sources to understand the data and fl ag potential issues? Is that tool integrated with the rest of the data quality and data integration platform? Monitoring and measurement. Does the platform enable you to establish key data quality metrics, monitor them on an ongoing basis, and receive alerts on items that fall out of acceptable ranges? Cleansing and remediation. Does the platform allow you to defi ne business rules to address data quality issues on automated basis? Does it provide historical statistics, which help root cause analysis on data quality issues, including accuracy, completeness, conformity, consistency, referential integrity, and duplication? Breadth of data. Does the platform include data quality capabilities that address all key data types customer, product/service, fi nancial, employee, etc. not just a single data type such as customer contact information? Ease of use. Does the platform provide an easy-to-use interface to enable both business users (e.g., business analysts and data stewards) and IT users to visualize and address data quality issues? Integrated metadata. Does the platform automatically capture the metadata from your data quality processes? Is the metadata seamlessly incorporated as part of the overall data integration lifecycle? 4. Data Consistency The platform should ensure that the value, structure, and meaning of data is consistent and reconciled across systems, processes, and organizations. Validation. Does the platform provide an integrated design and mapping tool that automatically validates the data model on-the-fl y? Transformation. Does the platform feature robust transformation capabilities that address not only syntactic issues, but also structural and semantic variances across different systems? Logical design and workfl ow. Does the platform capture all design and business rules at the logical level as metadata, abstracting them from the physical layer? Reusability. Are you able to capture all data integration and data quality logic and workfl ows as metadata via one platform? Does the platform enable sharing at both local and global levels? Cataloging. Does the platform allow you to easily search, fi lter, defi ne, and modify data dictionaries and business rules? Data synchronization. Does the platform easily interoperate with enterprise application integration (EAI) and messaging technologies to help synchronize the context and meaning of data, as well as data values, across operational systems? 6

9 White Paper 5. Data Auditability The platform should ensure that there is an audit trail on the data and that internal controls have been appropriately implemented. Lineage. Can the platform provide a visual lineage of data across multiple systems and applications, including both backward and forward tracking? Does it provide drill-down capabilities? Impact Assessment. Can the platform automatically assess the impact of changes across applications and systems? Does it provide reports on port details, metadata extensions and usage, and mapping dependencies across connected systems? Workfl ow. Does the platform include robust workfl ow orchestration capabilities including support for grid deployments and global, cross-team collaboration? Dashboard. Does the platform provide a dashboard with a high-level summary of workfl ows, processes, and status? Does it include the ability to easily drill down into the details? Testing. Does the platform include an integrated test environment that not only detects mapping and session errors, but also helps identify the root causes of invalid mapping and session errors? Version control. Does the platform feature robust, granular version management and deployment capabilities? 6. Data Security The platform should ensure secure access to the data. Data classifi cation. Does the platform support an enterprise-wide strategy on information classifi cation, enabling data to be easily grouped and classifi ed? Segregation of duty. Does the platform enable granular segregation of duties, as well as reporting on the interdependencies between different tasks? Privilege management. Does the platform have robust privilege management capabilities for managing and reporting on granular privileges such as copy object, maintain labels, and change object status? Client authentication. Can the platform integrate tightly with Lightweight Directory Access Protocol (LDAP)? Does it have a repository to track real-time authentication status? Web Services security. Does the platform support the latest Web Services security standards at both the message and transport layers for authentication, encryption, and authorization? Encryption. Can the platform support data encryption? Does it enable secure synchronization across fi rewalls, leveraging built-in encryption and compression capabilities? How to Launch a Data Governance Program 7

10 Architecture: SOA and Data Governance IT organizations should ensure that their data integration infrastructure is well-architected to meet the ongoing needs of data governance. Many IT organizations are adopting service-oriented architecture (SOA), and SOA has a signifi cant role to play in data governance. Conversely, data governance is critical to successful SOA implementations. To increase business agility through reusability of data assets, applications and processes must be able to access business-relevant data wherever it resides, in whatever form is required, when they need it, consistently and accurately. SOA, and in particular data services, are an excellent way to achieve this. In an SOA, data can be delivered as a set of shared data services, which are decoupled from specifi c applications and processes, and are managed in their own right. The data integration platform should provide a layer of abstraction via metadata, enabling data components and services to be reused without hand-coding. The platform should also publish the data services via open standards to allow easy interoperation with the rest of the IT architecture. Figure 2 shows how the Informatica data integration platform delivers data-related services, including data access, data integration and metadata services, to both internal and external consumers within the overall IT environment. Data Consumers Internal External Applications Databases Processes BI Tools Portals Applications Web Services Business Analyst Developer Tools Data Services Data Integration Services Universal Data Services Metadata Services Infrastructure Capabilities Administrator Data Sources Applications Databases Messages Flat Files XML Unstructured Data Figure 2: The Informatica Platform Architecture: Delivering Reusable, Shared Data Services Mainframe Just as SOA is critical to data governance, data governance is critical to SOA. Without a strong governance program, SOA becomes an incredibly fast, fl exible approach to proliferating data and data services across an organization with no control and no process. A data governance program that defi nes roles and responsibilities for data creation and access, enforces data standards and reuse, and establishes processes for sharing data will help deliver the promised benefi ts of SOA reuse of assets, standardization, fl exible deployment. 8

11 White Paper Organization: ICCs and Data Governance Organizational structures for data governance vary widely from company to company. The most successful governance programs are led by the business, with IT serving as a key collaborator in implementing an effective program. Another common element among successful data governance programs is the creation of an integration competency center (ICC), or center of excellence, within IT. ICCs are an organizational approach designed to increase agility and lower costs by creating a central pool of skilled resources, promoting reuse, sharing best practices, and establishing common processes and standards for integration. The ICC supports the data integration technology infrastructure and coordinates data integration projects across the entire enterprise. Within a data governance program, the ICC can as act as the primary interface between IT and the business. The ICC can defi ne the technical standards and processes around data governance and provide a pool of highly skilled technical resources who can support specifi c project and program implementations. The ICC staff work in conjunction with the data governance steering committees, data stewards, and data analysts to support their work and to ensure that the policies, processes, and standards they defi ne are systematically implemented in the data integration infrastructure. How To Get Started: Four Things Your IT Organization Can Do Now What can an IT organization do to support the launch of a data governance program? Here a few practical things you can do. 1. Tie the Pilot to a Key Business Goal When selecting a pilot, identify a project that will have a signifi cant impact on the business. Data governance requires strong involvement from the business tying the pilot to the success of a business executive sponsor will increase the visibility of the effort as well as help attract resources and budget. And after a successful pilot, having documented business value will facilitate efforts to expand the data governance program across the organization. Some common projects that can quickly deliver positive business impact are: M&A consolidation. If your organization has recently merged with or acquired another entity, there is often signifi cant pressure to integrate the organizations as quickly as possible. This includes integrating the data, as well as the systems. Data governance can help accelerate M&A consolidation by providing the framework for defi ning which data to integrate and how, aligning the different data defi nitions from the two entities, and assigning roles and responsibilities to specifi c people. And a robust data integration platform can signifi cantly lower the cost and risk of migrating data from one system to another during the consolidation process, a task whose complexity is often vastly underestimated and which often leads to project overruns and delays. Compliance. To ensure compliance with an ever-growing number of external regulations and internal corporate governance policies, companies must formalize the policies and processes around how they collect and manage data, and ensure accountability. A robust data governance program achieves just that. Supported by a data integration infrastructure that can automatically document the data and the data integration processes, organizations can create an audit trail to certify the validity and accuracy of data. A data integration infrastructure can also streamline the collection of reporting data, easing the burden of compliance. How to Launch a Data Governance Program 9

12 Master data management. Many organizations are implementing master data management initiatives to support key business goals. For example, they are creating a customer master to improve customer service and increase cross-sell rates, or integrating product master data to improve supply chain visibility and control. Regardless of the type of master data addressed, data governance has a critical role to play. Master data management projects inevitably raise questions regarding who owns the data, who gets to defi ne it, how it will be accessed and delivered, and which data standards will be used. A formal data governance program provides the organizational and process structures needed to successfully manage master data. 2. Make Pilots a Spreadsheet-free Zone When working on pilot projects, avoid the temptation to rely on the siloed tools your data stewards, architects, and analysts may use, such as Excel spreadsheets or stand-alone metadata modeling or data cleansing tools, simply because they are familiar. The goal of the pilot project should not be limited to developing the business standards and processes around data governance. The pilot should also be used to begin establishing the robust data integration infrastructure the organization needs to scale out the data governance program over time. Selecting a unifi ed data integration platform that meets the criteria identifi ed in this white paper will simplify deployment and help lower the cost and risk of ongoing data governance efforts. A pilot project presents an excellent opportunity to adopt new, more holistic technologies in a phased approach, allowing IT to work through any issues and gaps while still deployed in a limited scope. Demonstrating success through a pilot also eases the process of getting cross-functional buy-in to the notion of standardizing on a common data integration infrastructure. 3. Talk to the Enterprise Architects Make sure you sit down with the architects who own the overall architectural roadmap for the enterprise, particularly if these architects are not already intimately involved with the data governance program. It is important to understand their current architectural blueprint and future roadmap, and how they believe data governance plays into them. For example, architects may be setting technology standards that need to be taken into account in the governance program (and vice versa). In particular, a discussion around data services and how they fi t into the larger architecture will be critical to ensuring that the enterprise architecture is designed to support the goals of data governance. 4. Get IT Plugged into the Data Governance Structure Many data governance pilot programs designate a handful of data stewards and data analysts to lead the day-to-day pilot work, as well as establish executive steering committees or councils to provide oversight and guidance. It is important to formalize IT s involvement in and interaction with these data governance organizational structures from the very beginning. Certainly, senior IT executives should be participants in the steering committee or equivalent group. Also, IT organizations should begin examining the role of an ICC within the larger data governance structure. If an ICC already exists, that group should be involved in the data governance program from the outset. If no ICC exists, it is prudent to review the role that IT personnel such as data analysts, data architects, data integration specialists, and ETL developers are beginning to play in data governance, and consider centralizing at least some of those skill sets into an ICC. 10

13 White Paper Conclusion and Next Steps To derive maximum value from your data, you need a strong data governance program that helps you develop and manage your data as a strategic business asset. The success of your data governance program in good part hinges upon a robust data integration technology infrastructure that can support the processes, policies, standards, organization, and technologies required to manage and ensure the availability, accessibility, quality, consistency, auditability, and security of your organization s data. Developing the right technology infrastructure is critical to your ability to automate, manage, and scale out your data governance program. Now that you understand the limitations of applying siloed technology to data governance, the key technical requirements for a data integration infrastructure, and the role of SOA and integration competency centers in data governance, what are your next steps? Here are some suggestions: 1. Learn more about data governance by visiting: 2. Contact Informatica to fi nd out how your organization can put the Informatica data integration platform to work on your data governance program. Please call Informatica at About Informatica Informatica Corporation delivers data integration software and services to solve a problem facing most large organizations: the fragmentation of data across disparate systems. Informatica helps organizations gain greater business value from their information assets by integrating their enterprise data. Informatica s open, platform-neutral software reduces costs, speeds time to results, and scales to handle data integration projects of any size or complexity. With a proven track record of success that extends back to 1993, Informatica helps companies and government organizations of all sizes realize the full business potential of their enterprise data. That s why Informatica is known as the data integration company. How to Launch a Data Governance Program 11

14 12

15 White Paper How to Launch a Data Governance Program 13

16 Worldwide Headquarters, 100 Cardinal Way, Redwood City, CA 94063, USA phone: fax: toll-free in the US: Informatica Offi ces Around The Globe: Australia Belgium Canada China France Germany Japan Korea the Netherlands Singapore Switzerland United Kingdom USA 2008 Informatica Corporation. All rights reserved. Printed in the U.S.A. Informatica, the Informatica logo, and PowerCenter are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners (09/17/2008)