Data Quality Dashboards in Support of Data Governance White Paper
Table of contents New Data Management Trends... 3 Data Quality Dashboards... 3 Understanding Important Metrics... 4 Take a Baseline and Monitor... 5 Build a Business-focused Score Card... 5 Monitor Metrics over Time... 6 Market the Importance of Data Quality... 7 Tools to communicate... 7 Data Quality Dashboards... 8 Talend Data Quality... 8 Data Profiling... 9 Reporting... 9 Data Quality Portal... 9 Features... 10 Cleansing and Standardization... 10 Unified Data Management Platform...11 Page 2 of 11
New Data Management Trends In past years, you may have sought data quality tools in reaction to problems stemming from data failures. Your projects might have included a desire to improve revenue, mitigating a loss in productivity, or even ensuring the company was in compliance with industry laws and regulations. Even something as simple loading a target data source may have sent you off to purchase technical solutions to address the data quality problem or risk project failure. The responsibility was on IT to solve the problem and make the target application run right. Today, a newer trend is emerging when it comes to data management. More common today is a data governance approach where time and energy are spent on fixing the processes that allow flawed data to be introduced into an environment. In this approach, you rather seek to isolate the cause and eliminate the source of the introduction of flawed data. In companies where internal and external data is constantly being merged to form operational data stores and data warehouses, it makes more sense to not only address data failures, but to proactively monitor data and report the current state of data quality back to the origin of the data. This new data governance approach works on cross-functional problem solving and process improvement, not relegation of data management to IT. Data Quality Dashboards To this end, the emerging technology of data quality dashboards are becoming more and more important. Organizations pull together data governance strategies and these tools to: Understand what metrics are important to the organization Take a baseline of DQ metrics Monitor metrics over time Report the metrics so that cross-functional teams can understand the impact Page 3 of 11
Measure the impact and market the importance of data management You ll notice that the list did not include fixing data quality issues with tools, although this often remains a necessity. The emphasis in this approach is on modifying people s attitude and improving processes. The strategy is more about setting up data quality policies and setting up control processes based on data quality rules. The tactics are to automate monitoring and reporting of data quality metrics and communicate knowledge about the value of the data. The tactics include empowering business users with the ability to determine how best the data can be used to meet their own business needs. Understanding Important Metrics The first step in employing such a strategy is to understand key quality indicators (KQI) for your organization. Since every organization has unique data management challenges, you will have to reach out to your co-workers here and communicate. Communication is key to your success in data governance. If you were to sit down and work for a day in the billing center, call center or purchasing agent job, for example, people there will see that you understand them and care about their processes. The first, most important task you can do is to understand critical business processes in your organization. This will begin to develop your story about return on investment for justifying the funding of an improvement project, necessary data quality tools, master data management strategies and more. The most important metrics could revolve around any of the following key components: increasing revenue lowering costs reducing risks (compliance) Page 4 of 11
avoiding data disasters competing effectively in the market meeting the organization s objectives. Don t be afraid to ask questions, take notes and understand the data management challenges of your business associates. Take a Baseline and Monitor Having gathered potential data- quality metrics and business impact in the previous step, next it is time to take a baseline measurement. As part of the source system analysis, a baseline of each source system should be captured and stored as well as how multiple systems conform to expected metrics or business rules. In some cases, it will make sense to look not only at each source system in isolation, but across systems. Employ advanced profiling for comprehensive column and attribute analysis. Identify potential problems within structured data fields such as dates, postal codes, product codes, customer codes, addresses or any attributes that should conform to a particular format and structure. Configure custom data- quality rules, and flag any attributes that do not conform. When you re done with the analysis, you will have a very good idea of the challenges you face in integrating data and the information necessary to develop designs that address the challenges proactively. Build a Business-focused Score Card It s important to aggregate the information you get from a data profiling tool into business metrics. A simple non-aggregate score is usually rather meaningless. If we show that a certain percentage of all fields are nulls, for example, the number is of no use there is no context. You can t say whether it is good or bad, and you can t make Page 5 of 11
any decisions based on this information. Practitioners aggregate those scores into business value metrics. The technical metrics combined with the business pain observed allow business users and technical users to collaborate to govern data. Most data profiling tools and scorecards give you very technical metrics by default. More and more data profiling tools being developed today allow you to aggregate the technical scores into business metrics, but since a tool can t natively and unthinkingly know your business, it s up to you and the business stakeholders in the room to give it meaning. A common example of this might be to create an ability to contact score. Evaluate the completeness and validity of e-mail, name, address, and telephone number. Then create some simple logic around it that fits your business. It may be that having a name and address is enough to contact your customer. If an address is invalid, then both phone and e-mail are required by your data governance committee. Those records that have invalid addresses, and either an invalid e-mail or phone cannot be contacted by data governance rule. This ability to contact metric is easier to follow for your team. Monitor Metrics over Time Data may be coming into your databases via a call center or data entry. It may be coming into a centralized location from data feed or thirdparty vendors. Data quality dashboards allow you to keep an eye on data anomalies by constantly checking if the data meets business specifications. Dashboards offer attractive charts and graphs on the status of data compliance. It shows your key performance indicators and trends conformity. Today s modern businesses use data quality dashboards to monitor and report feeds from the company s many locations. When you want to ensure that client information was properly being entered into your database, profiling it and comparing it to the baseline. With Page 6 of 11
dashboards, your headquarters could report back to the individual data stewards and notify them when they get the data right and when they had allowed anomalies to get into the system. This feedback loop works to continually improved data quality, train data stewards at new locations and improve billing efficiency. Dashboards are also a great way to make your people smarter about data governance; they will help you achieve cleaner, more useful information. Market the Importance of Data Quality It s unfortunate, but people have the tendency to think that responsibility for information quality starts with someone else, not themselves. In truth, we all know that information quality is the responsibility of everyone in the organization, from the call center operators to the sales force to IT and beyond. Tools to communicate Tools that supplement your in-person meetings can track progress, promote the power of data governance, help you take on difficult challenges and keep a record of successes. The good news is that there are some fantastic software tools that can support communication. Workflow - With workflow tools, teams can manage the processes and coordination of the data governance team. Wikis - In your data governance projects, you can use a Wiki to document processes in a data governance projects, and have an open online dialogue about risks and challenges in the project. Blogs - A blog allows for one person s perspective on the data governance project, but readers can leave comments and links to their own blogs. Blogs can educate and inform data governance groups, and they can use them to debate unresolved issues or to continue discussions between meetings. Page 7 of 11
Data Quality Dashboards Knowing where your organization stands with regard to data quality is essential for running an efficient and profitable business. Talend Data Quality is designed to allow companies to share the vision of data quality across the entire company. In addition to its profiling, cleansing, standardization and matching functions, Talend Data Quality provides customizable, web-based data quality reports and scorecards as well the ability to export them in popular formats like Adobe Acrobat, Microsoft Excel and XML. Business users and IT users can team up in formulating processes that keeps data clean in the enterprise. It extends the dynamic reporting capabilities of Talend Open Profiler to help organizations monitor data quality metrics and support data governance initiatives. Talend Data Quality With Talend Data Quality, organizations have a single, powerful solution for reporting and monitoring the quality of their data across the enterprise, allowing them to easily align the data quality facets of a data governance program. This alignment improves processes and communication on your valuable data assets. With Talend Data Quality Page 8 of 11
data quality scorecards can be distributed to the data source owners to track data quality improvements over time and drive compliance with enterprise-wide data quality standards. Data Profiling Data profiling provides both an evaluation of your current state of data quality and measurement of it over time. You can generate informative reports and share key data quality metrics with your team the first step in solving data quality issues. Reporting Talend Data Quality includes report generation where data stewards can leverage the data profiling results to create predefined reports that watch for the violation of data quality thresholds. The reports can be generated in Adobe Acrobat and other formats and communicated via e- mail, wiki or intranet site. Data Quality Portal Talend Data Quality delivers customized, key quality indicators to a web-based portal where teams can collaborate on the process of Page 9 of 11
improving data quality across the enterprise. Data Quality Portal provides an easy-to-use, browser-based view into key data quality metrics. The portal opens up the issue of data quality to a broader audience in your organization, fostering positive change around the way your company manages data. Features Create customizable web-based dashboards that display critical information on data quality processes needed by business users Provide personalized, default dashboards for different levels within the organization that include alerts, data views, and links Access control so that users and managers only see the dashboards they need for their job Alerts that update in real time so you re always on top of data quality issues Track progress against realistic targets on an attribute-byattribute basis Export reports in popular formats like PDF, XLS, XML and HTML. Cleansing and Standardization If you need it, Talend Data Quality has powerful tools for repairing and cleansing data as well. Talend Data Quality allows you to use internal or external reference data to set the standards for values, regular expressions to set standards for data shape and size. Set up cleansing processes using a wide range of dedicated data integration and quality components. Nicknames or addresses are for example easily normalized and standardized using the dedicated components. These dedicated components, such as name & address cleansing components, fuzzy deduplication components, third-party Page 10 of 11
address validation & standardization services are natively available in Talend Data Quality. Unified Data Management Platform Talend is an open source data management company offering complete, end-to-end data management tools. Data Quality is the foundation for building your complete data management strategy, but data management is not limited to Data Quality. Data management requires that you be able to access the data no matter what source system, manage metadata and provide the transformations that come with Data Integration. Data management is also about process change, the kinds of changes you make as you move toward Master Data Management and a more mature data governance strategy. Talend tools provide an easy path as your data management needs mature. Page 11 of 11