A DataFlux White Paper Prepared by: Mike Ferguson Data Ownership and Enterprise Data Management: Implementing a Data Management Strategy (Part 3) Leader in Data Quality and Data Integration www.flux.com 877 846 FLUX International +44 (0) 1753 272 020
Companies need a strategy in place in order to get under control and manage it on an enterprisewide basis. Introduction So far in this three-part series on ownership, I have discussed what ownership is, why it is important, what the key requirements of enterprise management (EDM) are and how companies can address the management problem by standardizing on a suite of technologies, which I referred to as an EDM suite. In this, the third and final paper in this short series, I want to look at what needs to be done from a strategy perspective to be able to establish personnel and procedures for enterprise management, and what needs to be done in order to leverage the technologies available in an EDM suite to get maximum return on investment. Enterprise Data Management Strategies In the first paper of this series, we outlined three key requirements for enterprise management. These requirements are: Establish a common suite of technologies for end-to-end management Dedicate IT personnel to enterprise management Establish policies for governance Having looked at the first of these already in the second paper, we now turn our attention to organizational structure and governance concepts that are fundamental to any management strategy. A Chief Data Architect should have extensive experience in the vertical industry that he or she works. Organizational Structures for Enterprise Data Management One of the key appointments any company can make to help get their under control is the position of a Chief Data Architect. This is often a position overlooked in IT and sometimes not well understood by business. If it does exist, this person must have a business mandate to cause change so that can be brought under control. Fundamentally, the job of a Data Architect is to understand how is used in business on an enterprise-wide basis and to formally define the used. This individual is also responsible for setting policies and procedures for the use of that, for maintaining quality, and for ensuring a common consistent understanding of what means. Ideally, a Data Architect should have extensive experience in the vertical industry that he or she works so that they can clearly discuss in the context of its business use. Data Architects must also have expertise in management skills such as: Implementing standards and establishing policies for developers and business users, including defining standard enterprise-wide vocabularies In-depth understanding of the relational model and navigating XML schemas Data modeling and modeling techniques such as normalization and star schema multi-dimensional modeling, as well as some fluency in the use of modeling tools Logical and physical base design Data profiling and defining rules for content cleanup
Understanding of the requirements that regulations and legislation impose on for the purposes of compliance The architect needs an enterprisewide business mandate. Ideally, architects should have an enterprise-wide remit in the sense that they need to operate across all lines of business when managing. This is especially important in setting strategy and patterns (best practices) around specific management processes such as: Master management Data profiling and monitoring Data migration and consolidation Data replication and change capture Data synchronisation Data federation Data warehousing and aggregation Data security Taxonomy design It helps if enterprise management IT professionals can work with otherbusiness integration professionals in an integration competency center. Many companies are starting to create centralized IT expertise in business integration by creating Integration Competency Centers so that IT professionals responsible for different types of integration are able to coordinate their work. The architect is at the center of management, quality and integration and should be a key member of any integration competency center initiative. Figure 1 shows five levels of business integration. Data and meta integration (and management) underpin and are a key piece of any business integration initiative.
Organization Structure EDM As Part of An Integration Competency Center To Coordinate Integration ETL, EII, DQ, master management & content management EAI and SOA integration platforms Business process management software Collaboration tools Enterprise portal software Strategic Objectives Integration is happening at all these levels Data and Meta Integration lication Integration Business process integration People integration User interface Integration e.g. Reduce operational costs Co-ordinate integration to achieve an objective Co-ordination requires an Integration Competency Centre Figure 1 - Five levels of business integration Figure 2 shows how such an enterprise management team operates. The first thing to notice is the consolidation of IT professionals responsible for management and integration in operational, BI and unstructured content management systems into a single team. This enterprise management team includes the users of the EDM suite of technologies, and this team has a responsibility to set standards and support the other IT professionals working in specific lines of business throughout the enterprise. People in this team are EDM technology experts. EDM in An Integration Competency Centre - A Federated Organisational Structure Is Worth Considering Companies may benefit from merging operational, business intelligence and content management integration IT professionals into one team. Executive Corporate Integration Competency Centre Business units Dedicated EDM team in the ICC Merge operational, BI & content management IT integration teams sponsor Data Governance Steering Committee Enterprise Data architects Data naming and definition standards Enterprise Data Model Data Security Data integration development and management policies Master management Taxonomy design Integration with other technologies Content management, community taxonomy maintenance, modelling using common definitions, integration templates Figure 2 - How an Enterprise Data Management team should operate
As an example, if IT professionals in different lines of business wish to create models then these models would be constructed from standard definitions and entities made available to them by the EDM team. The same applies if there is any development of. It is also important that such a team is backed by an executive sponsor who participates with a governance steering committee. At the risk of being criticized for suggesting yet another steering committee, I would at least argue that in today s climate of much tighter regulations, CFOs are the likely sponsors as they take compliance very seriously. There also appears to be no shortage of executives lining up to participate in such a steering committee. Having a Compliance Manager on such a committee is also important. Enterprise Meta Management and Integration Another key element of an enterprise management strategy is governance. This is about defining standards including common names and definitions (common meta), common policies, patterns and processes for enterprise quality and integration development, the construction of an enterprise model, and defining policies and processes for master management (MDM). A shared business vocabulary of common names, definitions and integrity rules needs to be established. Data standardization requires that a shared business vocabulary (SBV) is established. Setting up a shared business vocabulary involves identifying and defining used in the enterprise, creating a common set of enterprise-wide standard definitions for that and then (cross referencing) disparate definitions to these common definitions. More specifically, the SBV involves incrementally defining a set of enterprise-wide common names, common definitions, common integrity rules, common reference (e.g., code set values), common s and common transformations for all master, transactional, dimensional and metrics. The SBV forms the base of an enterprise-wide standard. It is fundamental to the success of commonly understood, master management and integration. These common definitions can be used in: Data models to get consistency across multiple models Data integration tools (ETL and EII) lication integration technologies (message brokers and ESBs) Business views of reporting tools XML mark-up tags Rendering (e.g., in XML form) using standard XML tags based on SBV names means that can be presented using names that are commonly understood by users. In addition, if is made available for consumption by applications in the same way, then the is managed in a consistent unambiguous fashion as it travels throughout the enterprise and as it is prepared for presentation. To explain why this is important, consider Figure 3. This shows a common problem that often arises when trying to integrate Performance Management products with multiple lines of business intelligence systems to calculate key enterprise level performance metrics.
Why Do We Need An SBV? - How Do You Drill Down From CPM Products When Metrics Definitions Are Inconsistent? CPM product Revenue? metrics definition Total Revenue KPI metric definition DBMS Packaged analytic app n BI Tool DBMS meta Custom built mart Total Sales? metrics definition n Packaged meta analytic app mart Turnover? metrics definition BI tool Custom meta built mart Common definitions are critical to BI integration Figure 3 - SBVs better integrate Performance Management products with multiple lines of Business Intelligence systems It is difficult to integrate BI systems at the enterprise level when they have inconsistent names and definitions. If each underlying BI system has been built independently, then what happens when you have three different metrics in three different BI systems called Revenue, Total Sales and Turnover, and you want to create a Key Performance Indicator called Total Revenue? Do you think business users understand the difference between these metrics? Worse still, do you think an IT developer knows the difference? The problem here is obvious. It s ambiguous and prone to misunderstanding. This misunderstanding can lead to erroneous reporting and opens up the door for potentially incorrect interpretation of and incorrect decision making. A best practice would be to prevent this and to establish a shared business vocabulary across all BI systems and business views. This is done in some companies but not necessarily in all. Nevertheless, standards are about preventing different development teams inadvertently introducing ambiguity. Even if common definitions are practiced across BI systems, it is very likely that the same could not be said when you head into the world of operational systems. Most operational systems have their own application-specific names and definitions (application-specific vocabularies) for that they maintain. Therefore, when you consider Figure 4, it is not difficult to see the problem caused when integrating applications directly into enterprise portals, for example, to integrate and simplify user interfaces. One important question to ask when looking at Figure 4 (below): What is the problem with this architecture?
Why Do We Need An SBV? - Plugging lications Into A Portal Means Each lication Displays Its Own Data Names WebServices \ SQL \ Custom Portlet Portlet Portlet Figure 4 - SBVs provide consistency across multiple applications. Integrating the user interfaces of disparate applications in a portal will highlight all the differences in disparate names across all applications. The answer, of course, is obvious. All applications present their using their own application-specific names (vocabulary). If the same is used in different applications (e.g., customer, product, order transactions, etc.) and each of these applications use different names for the same, then the user has to know this to correctly understand presented to them on a portal page by the different applications. Worse is when different in different applications have the same names and this is presented to the user on the same portal page. In this case the user once again must be aware of the differences if they are to accurately understand what they are examining. In order to resolve this problem, any application-specific rendered using application-specific XML tags for presentation on a portal page needs to be intercepted and translated into commonly understood names before the user sees it. This can be achieved by introducing a message broker or enterprise service bus (ESB) technology between the application and the portal. When this happens, marked up using application-specific XML tags can be translated at run time into common XML tags when the appears on the screen using SBV definitions. Simply put, the application-specific definitions still exist but have been hidden by the introduction of message-broker or ESB software. Similarly, if message-brokers or ESB technology is used for application integration in a service oriented architecture (SOA), then in any messages that travel between application services as part of a business process needs to be translated from source application specific mark-up to common mark-up, and from common mark-up to destination application specific mark-up. What does all this mean? It means that business integration software needs to make use of SBV common definitions and s from disparate systems to common definitions. Is this familiar? It should be and to show you why, look at Figure 5 (below). Figure 5 shows a remarkable coincidence when comparing application integration software (message brokers or ESBs) and on-demand integration technology
(sometimes referred to as enterprise information integration, or EII). EII products, part of an EDM technology suite, present virtual integrated views of disparate in multiple underlying systems and allow these virtual views to be accessed as if the was integrated in a base. These virtual views can be defined using common standard definitions (i.e., using the SBV definitions). Data integrated in real-time by EII products will then render the marked up using common definition tags. Data integration technologies also need to make use of a shared business vocabulary. Common Meta Is Needed In Data and lication Integration Technologies To Achieve Consistency all presented in common vocabulary in the portal web service adapter EAI lication Integration Common vocabulary platform WSRP EII Data Integration Composite service Composite service web service adapter common vocabulary integrated virtual view vocab 1 vocab 2 vocab 3 vocab 1 vocab 2 vocab 3 EII works by giving applications an on-demand virtual integrated common vocabulary view of disparate Figure 5 - Understanding why business integration software needs to make use of SBV common definitions and s from disparate systems to common definitions These are the key points to remember: Data s from disparate to common are also needed by multiple technologies. To integrate when building a warehouse using ETL technology, you need common definitions for the target system, and you need to know how disparate in source systems maps to common definitions To integrate on-demand (e.g., for reporting or presenting on a portal screen) using EII technology, you need common definitions for the integrated virtual view of the and need to know how disparate in source systems maps to common definitions in the virtual views To manage electronic messages as they enter the enterprise and move between applications, message brokers and ESB software, you need to know the common definitions for and how disparate in source application systems maps to these common definitions so that message translation can take place To achieve consistency across multiple performance management and reporting tools, you need common definitions for the BI and performance management tool business views, and you need to know how disparate in source systems maps to the common definitions in the business views
To define master and solve the master management problem, the master entities to be defined using common definitions and then the s from disparate to master and vice-versa, you need to define this so that master integration and synchronization can be managed The secret to enterprise management is being able to share common meta across multiple technologies to achieve consistency. In fact, everywhere you look you see precisely the same requirement again and again. The secret to Enterprise Data Management is therefore in the meta. If you first capture the shared business vocabulary and all the s from disparate definitions to common SBV definitions, then that meta can be provided to and shared across: EDM suite technologies, such as ETL tools or EII tools, Data modeling tools for building models using consistent common definitions MDM applications and technologies BI and Performance Management business views Message brokers and Enterprise Service Bus technologies used in application and business process integration Portal technology to present the for business use. Combine this with enterprise quality and the quality firewall discussed in the second paper in the series, and you can see the whole strategy for enterprise management coming into clear focus and taking real shape. Figure 6 shows the power of common meta when you have an SBV and know the s from disparate definitions to common ones. If you have the tooling to do this work once, all the consistency needed to manage across the enterprise stems from the same base meta, as long as this meta can be shared across technologies.
Data Governance - Use Created Common Meta To Generate Mappings For Multiple Integration Technologies To Achieve Consistency all presented in common vocabulary in the portal web service adapter EAI lication Integration Common vocabulary platform C master R asset prod cust D WSRP U historical DW mart mart mart Composite service Composite service web service adapter common vocabulary integrated virtual view EII Data Integration platform Operational vocab 1 Operational vocab 2 Operational vocab 3 Enterprise DQ & Data Integration vocab 1 vocab 2 vocab 3 Generate common vocabulary & XSLT s Common meta Generate common vocabulary virtual model and s Figure 6 - Data governance and the power of common meta Enterprise Data Quality In addition to establishing a shared business vocabulary, an EDM strategy involves the establishment of a quality firewall to validate entering the enterprise via keyboard, electronic message or file. Implementing profiling technology and quality as a service so that it can be invoked on-demand, in-batch, on a timer-driven and event-driven basis is the way to handle this. Enterprise quality technology is a key piece of the EDM suite of technologies outlined in the second paper of this series. Common rules defined for a shared enterprise quality service is vitally important to consistent validation, repair, matching and handling missing. A shared business vocabulary, EDM technologies and a quality firewall are all needed for successful MDM. Master Data Management Master management involves using the technologies in the EDM suite to solve the master management problem. Key business entities such as Product, Asset, Employee, Customer, Supplier, etc. need to be defined using the common definitions of the shared business vocabulary. Data integration technologies leveraging the SBV and s from disparate to the SBV definitions can then integrate master from disparate line of business applications and persist it in a master hub. In addition, the same meta also allows MDM solutions to synchronize subsets of master used in operational applications when master is updated centrally, and supplies dimensional to warehouses and marts. The quality firewall protects the quality of master and manages all changes to it via the keyboard, electronic message and batch files.
Organizational structure, EDM technologies and governance are the key pieces needed to get control of your and achieve compliance. Conclusion An enterprise management strategy involves getting the organizational structure right, selecting technologies that form an integrated EDM suite for handling all meta management and management needs, putting controls in place, and setting up governance processes. These things together allow the enterprise to take control of ownership, achieve compliance and raise the bar on quality, business practice and business confidence. The keys to this EDM strategy require: a shared business vocabulary; meta integration and meta sharing across enterprise integration technologies; a quality firewall; and master management. We have the pieces to solve the problem. Establishing a strategy for enterprise management will help you take back control of the in your enterprise.