SQL Server Master Data Services A Point of View

Transcription

1 SQL Server Master Data Services A Point of View SUBRAHMANYA V SENIOR CONSULTANT SUBRAHMANYA.VENKATAGIRI@WIPRO.COM Abstract Is Microsoft s Master Data Services an answer for low cost MDM solution? Will it provide enough capabilities to address master data requirements of an enterprise in its first edition? Read this article to know more on this. The article gives an MDM architect s perspective on Microsoft s MDS to address different master data requirements of an enterprise. It provides a comprehensive assessment of the tool on different functional and non-functional MDM dimensions. The findings presented here are based on author s experience with the tool in a proof of concept project involving multiple MDM tools. 1

2 TABLE OF CONTENTS Introduction... 3 Parameters Considered for MDS Capabilities Assessment... 4 Our View Point of MDS along MDM Dimensions... 6 Data Repository... 6 Data Integration... 6 Data Quality... 8 Metadata Management... 9 Data Management... 9 Data Governance Architecture & System Management Concluding Remarks ABOUT

3 INTRODUCTION Microsoft has introduced its Master Data Services (MDS), a Master Data Management tool, in its SQL Server 2008 platform which was officially released in the middle of With this release, Microsoft has made an official entry into MDM space as a tool vendor targeting mainly SME customer segment. Microsoft had made a strategic acquisition of Stratature, predominantly a Hierarchy Management tool and MDS is the new avatar of this Stratature with Microsoft investing good amount of time and effort in integrating this acquired solution into its SQL Server product stack. It is making all efforts to best integrate this acquired technology with the Microsoft product line to maximize customer value and reduce functional overlaps with its other products. As published by Microsoft, the first delivery of MDS will focus mainly on analytical hierarchy management that potentially can provide business executives the ability to create and manage existing and custom data hierarchies. Microsoft has recently made its CTP Nov 09 version of SQL Server 2008 available to some of its partners and customers that includes Master Data Services as an additional component. The intention of this release was to evaluate the tool capabilities in real world customer scenarios, learn about customer expectations from the tool from features perspective and get feedback from customers and partners. The purpose of this document is to present a perspective on Microsoft s Master Data Services tool s capabilities to address master data management requirements of an enterprise. This view is primarily based on author s involvement and experience in an evaluation exercise of MDS conducted for an Energy industry customer. 3

4 PARAMETERS CONSIDERED FOR MDS CAPABILITIES ASSESSMENT In general, tool s capabilities evaluation exercise involves assessment of its features richness and completeness against all functional and non-functional requirements that are expected to be fulfilled by the solution that uses the tool as the basic building block. Generic evaluations can at best serve as starting point in selecting MDM tools. Such evaluations lack the context of organizational requirements - the key in a MDM tool evaluation. To address this vital requirement in tool evaluations, Wipro has built a Product Capabilities Evaluation Framework (PCEF) specifically for assessing the MDM tools. Wipro s MDM Product Capabilities Evaluation Framework defines multiple conceptual dimensions which comprehensively addresses all master data management requirements of an enterprise. Each of these dimensions defines a group of requirements that needs to be satisfied by an enterprise level MDM solution. We have used these MDM dimensions to assess the capabilities of Enterprise SQL Server s Master Data Services. Since Microsoft considers MDS as a feature component of Enterprise SQL Server and is meant to leverage SQL Server Integration Services (SSIS) wherever possible, we have made a conscious decision to include the support capabilities of other tools (such as SSIS, SSRS, Service Broker) of SQL Server stack for the purpose of this evaluation. Following MDM dimensions were considered to arrive at this point of view - Data Repository This dimension explores data modeling capabilities of an MDM solution to model multiple and complex master data domains, associated entities and their inter-relationships, hierarchies, etc Data Integration The ability of MDM solution to integrate with different source and subscribing systems both in batch and real time modes are evaluated using this dimension Data Quality This dimension captures the capabilities required for maintaining high quality master data in the MDM solution particularly focusing on data standardization, de-duplication, match and merge, data enrichment from external sources, ability to maintain source data references Metadata Management This dimension examines MDM solution s ability to manage business, technical and process metadata Data Management This dimension attempts to assess the capabilities around content authoring and editing and its associated security, search functions, workflow support for master data quality control, etc Data Governance The ability to manage data governance policies, practices/procedures and their enforcement though workflows and security features of the MDM solution is assessed by this dimension Architecture & System Management This dimension examines the MDS solution s ability to support different styles of MDM, security aspects in terms of Single Sign On, integration capability with enterprise LDAP, MDM s performance & scalability aspects, general server administration capabilities such as monitoring, error handling, backup & recovery, ability to migrate configurations and data between environments 4

5 As mentioned earlier, Microsoft is in the process of reducing capabilities overlap across its Enterprise SQL Server tool stack and recommends the usage of complete SQL Server capabilities (to supplement MDS features) for any MDM implementation. Hence, we are treating all support capabilities of SQL Server tool set as part of MDM capabilities in this evaluation. This overall MDM capability of SQL Server is henceforth named as MS MDM Platform in this document. The following section attempts to assess MS MDM Platform capabilities with reference to aforementioned dimensions and provides our point of view on each of the MDM dimensions. 5

6 POINT OF MDS ALONG MDM DIMENSIONS DATA REPOSITORY KEY CHARACTERISTICS The model in MDS is metadata driven and hence a business user can model the master data entities and their relationships using the standard UI provided. MDS comes with no pre-built repository for any master data domain and is domain neutral. It provides good support for modeling parent-child / hierarchical, lookup type of relationships (one to many cardinality). It also supports modeling of many-to-many relationships through relationship entity, but requires the user to be proficient in relational data modeling concepts. It does not provide support for cross domain/model relationships. Strengths Provides a simple UI for data model management with ability to model most relational data modeling constructs Domain neutral with no default domain / data model at creation Good support for defining look up and hierarchical relationships Ability to create custom hierarchies and excellent UI capabilities for hierarchy management Weaknesses No capability to model cross domain relationships Limited navigation capability through UI for many to many relationships No support for multi-valued attributes, attribute inheritance across nodes in hierarchies Limited support for unstructured content management MDS is a good candidate where data domains are isolated and do not have inter-relationships. It provides sufficiently good support for modeling relationships between entities within a model. Its suitability for complex master data domains with high data volumes and where there is a requirement for unstructured content management needs to be evaluated through pilot implementations. DATA INTEGRATION KEY CHARACTERISTICS MDS has 3 pre-defined staging tables to support bulk load of master data into its repository (one each for entity, attributes and relationships). Data from source systems have to be massaged, transformed to the required format using 6

7 SSIS (any other ETL tool) before loading to staging tables. There is no support for real time or near real time data transfer from source systems. Data syndication from MDS is limited to exposing subscription views (data base views) of entities in SQL server database. It does not provide inbuilt support for delta load or delta export. Strengths Simple, unified staging table structure for bulk data loads Provides capabilities to bulk delete (de-activate) and bulk update objects in MDS repository Web services support for real time integration Export is based on subscription views which are basically database views and hence are highly accessible by most subscribing systems Weaknesses No MDS support for extraction, transformation and loading of data into staging table, requires processes to be created in SSIS tool Source data needs to be un-pivoted leading to huge amount of data in staging tables Could take long time for large data loads Error reporting is onerous and not very useful at times No delta detection mechanism for delta loading & exports No unified UI for overall data load process. Requires lot of work to be done in SSIS No support to run business validations before bulk loads (No staging area for data quality checks prior to load) No out-of-the-box support for exporting data in standard formats. Relies on ETL jobs created in SSIS Limited messaging based integration support Bulk data loading into MDS is cumbersome and it is a time consuming task as data from source systems have to be unpivoted, transformed before loading into staging tables. These tasks have to be done outside of MDS. Moreover, SSIS is not integrated into MDS leading to the usage of multiple tools / development environments / UI s to accomplish bulk load task. Bulk load function may not scale for large volumes of data specifically for entities with large number of attributes. Similarly, data exports in any given format (such as XML, CSV, etc) have to be performed using SSIS and since MDS is not tightly integrated with SSIS, this task requires different skill set. In our view, the current features provided by MDS for bulk load and export are very primitive. Microsoft should tightly integrate MDS with SSIS and provide a unified user interface in MDS for bulk data loads and exports. It is also recommended to include additional features to support delta detection mechanisms for delta loads and delta exports. 7

8 DATA QUALITY KEY CHARACTERISTICS MDS provides capabilities to configure and execute business rules to control the quality of data loaded into its repository. It has a simple business rule builder to configure business rules on entities and attributes. The validation status will be set appropriately on each record when these rules are executed on the records. There is no out-of-the-box support in MDS for any other data quality enforcement functions such as data standardization, de-duplication, data enrichment through connectivity to external data sources, match and merge, etc. Strengths Provides a simple business rule builder to create business rules to control the data quality with ability to assign priority for execution Ability to exclude certain business rules from execution Provides facility to define cross entity validations and complex validation rules Business rules can be executed from external application through web services Weaknesses Validations cannot be enforced at data model level (such as required attribute) No features in MDS to support de-duplication, data standardization, match (exact and/or probabilistic), merge & unmerge, data enrichment of master data records. All these data quality processes have to implemented using available functions of SSIS No integration support with external DQ tools Primarily supports Load and Validate mode of data quality process as there is no staging area available for data quality processes No in-built support for maintaining references to source data records. This has to be built explicitly into the data model and managed MDS is heavily dependent on SSIS for most data quality process implementation. Only business rule validations can be configured in MDS. With SSIS supporting MDS, data standardization, matching functions can be implemented. But, functions such as merge, survivorship at attribute level for merged records require lot of custom development. One of the major deficiencies of the MS MDM Platform is its lack of support for integration with external DQ tools. So, in our view, the data quality process support from MS MDM Platform is insufficient at this point of time. 8

9 METADATA MANAGEMENT KEY CHARACTERISTICS MDS provides a deployment utility which provides capability to export metadata created in an MDS repository. The extracted metadata can be imported into a new repository using the import metadata function. However, security and subscription view metadata cannot be exported. Although some of the advanced metadata management features such as managing business glossaries, process related metadata such as review policies, etc are missing, MDS enables management of technical metadata (models, configurations, etc) which is sufficient for most MDM implementations. DATA MANAGEMENT KEY CHARACTERISTICS MDS provides an explorer function for data management through user interface. CRUD operations on master data can be controlled by configuring security in MDS. All changes made to master data records in MDS are logged and changes can be rolled back through undo operation. It provides good user interface to browse and manipulate hierarchy data. It also has good search capabilities which includes probabilistic search. However, there are few deficiencies in MDS w.r.t. data management. It neither provides UI based bulk-edit functionality nor there is any in-built support for data management workflows or entity level version control mechanism. Strengths Good UI for CRUD operations on master data objects Ability to control access on master data management operations at attribute level with security features Ability to trigger external workflows for data enrichments & data quality control Excellent search capabilities with support for exact and probabilistic searches including hierarchy data Ability to log each change on the master data element and feature to reverse changes if required Weaknesses No built-in workflows for data management, but has ability to trigger external workflow processes It does not provide concurrency control mechanism (such as record level locks) and state management which are essential for DM workflows No version control mechanism at entity level, entire model with data has to be versioned No bulk update is possible using the standard UI provided Capabilities to export hierarchy and other master data 9

10 into excel Web services support for custom UI development Thin (web browser based) UI MDS has adequate data management functions which can be leveraged to manage master data for specific data domains where solution is used mainly for operational and analytical purposes. Since it lacks workflow and concurrency control features, it may not suitable in situations (such as product domain) where there is a strong need for collaborative data creation and management requirements. DATA GOVERNANCE KEY CHARACTERISTICS Data governance in MDS is limited to setting up access control to different data elements in its repository. It has no inbuilt workflow capabilities, and hence relies heavily on external workflow engines to support any data management workflow requirements. Microsoft has plans to develop add-on components in their future release to support integration with its Share Point workflows. Data governance, in general, is not adequately supported by most MDM solutions available in the market today. It is related to the definitions of processes, policies/practices and enforcing compliance to these processes and practices within enterprise to control quality of master data. Tools can only enable the enforcement of these data governance processes and practices to some extent through access control and workflow mechanisms. In this context, MDS although provides security features to control data access, has limited support for the required workflow processes. In our opinion, there is scope for improvement for MS MDM platform in data governance area. ARCHITECTURE & SYSTEM MANAGEMENT Architecturally, MDS can support analytical and operational MDM styles more effectively than the collaboration style as MDS lacks support for workflows. MDS is more suited for consolidation and/or centralized architectural styles of implementation. Registry style may not be very well addressed by MDS as there is no support for this style of implementation and has to be built into MDS installation at data model level. 10

11 From security perspective, while MDS provides integration with active directory, its integration capabilities with enterprise LDAP is yet to be tested. The database behind the MDS is completely exposed to other users and MDS has no means of securing access to its database from external users and systems. Data load performance from MDS is debatable specifically in large implementation with large number of entities and attributes and huge data volumes as all incoming data needs to be un-pivoted and loaded into staging tables resulting in multi-fold increase in the overall data load volumes. User interface seems to perform adequately well specifically for search operations when master data in the repository is limited to less than a million records. Microsoft is yet to publish any benchmark figures on MDS performance and hence it may be too early to comment on MDS performance. MDS provides capabilities to archive the entire repository along with its contents and deploy the same in another environment. This is an important requirement from an MDM solution where configurations with or without data need to be migrated from development to test to production environments. However, incremental repository migration is not supported which may be a concern. 11

12 CONCLUDING REMARKS With MDS as part of its Enterprise SQL Server stack, Microsoft is making a late entry into the MDM market where there are already many established vendors with wide implementation experience. However, most of the available MDM platforms in the market are a little too expensive for SME customers and many of the features that are provided by these MDM tools may not be of much relevance to customers in this segment. The positives of MDS as compared to its competing products are its ability to provide a low-cost MDM solution which is domain neutral with good data modeling capabilities, easy configurability with good hierarchy management features. Its ability to leverage earlier investments of enterprises in SQL Server and other Microsoft products also works to its advantage. However, since it will be the first version of the product (released during mid of 2010), its ability to scale, perform at an enterprise level needs to be tested with pilot implementations. The stability of the platform will also be unknown with the initial release of the product. Moreover, MDS is best suited for either transactional and/or analytical MDM implementations while it is not recommended for collaborative MDM needs. It may not be suitable for registry-style of MDM implementations either. To conclude, Microsoft s MDM platform is best suited for low cost, low to medium volume MDM implementations particularly for Small and Medium sized Enterprises. However, since it is still a new product, enterprises are advised to take cautious step to first use this product in a pilot implementation to assess its performance, scalability and usability for the enterprise before embarking on full scale enterprise level implementation. 12

13 ABOUT Wipro is the first PCMM Level 5 and SEI CMMi Level 5 certified IT Services Company globally. Wipro provides comprehensive IT solutions and services (including systems integration, IS outsourcing, package implementation, software application development and maintenance) and Research & Development services (hardware and software design, development and implementation) to corporations globally. Wipro's unique value proposition is further delivered through our pioneering Offshore Outsourcing Model and stringent Quality Processes of SEI and Six Sigma. Copyright Wipro Technologies. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without express written permission from Wipro Technologies 13