Data Governance and Management
Personal background combines technical, human and healthcare perspectives Almost 15 years IS research and development University of Turku, Finland Information systems Empirical field studies in healthcare settings Turku University Hospital, Finland Healthcare datawarehousing Project management, system and service design. Aalto University, Helsinki, Finland Usability research Healthcare data and information quality research Siili Solutions, Helsinki, Finland Information management consulting Sami Laine Data Science Architect, Siili Solutions Oyj, Finland
Organization that uses enterprise data in creative ways to unlock business value and beat competition Data-Driven Organization
Data-Driven Organization In order to achieve this, we need proper Data Management. And remember: Data Management is not only an IT driven initiative.
The Definition of Data Management Development, execution and supervision of plans, policies, programs and practices that control, protect, deliver and enhance the value of data and information assets. Source: DAMA International: Data Management Body of Knowledge (DMBOK)
Data Governance explained Ensures that data meets the expectations of all business purposes, in the context of data stewardship, ownership, compliance, privacy, security, lifecycle and quality. Data Governance Siili Oversight Provides formalized discipline to ensure accountability for the management of company s core information and provides structure and sponsorship for decision making. Siili Policies and Standards Processes and Best Practices Well defined governance structure guarantees that data can be trusted and that people can be made accountable for cases where poor data quality or mislead processes lead to adverse events. Requirements and Change Mgmt Issue Resolution Siili Data Quality Metrics and Monitoring
Data Governance explained Data Governance Siili Oversight Siili Policies and Standards Processes and Best Practices Requirements and Change Mgmt Issue Resolution Siili Data Quality Metrics and Monitoring
The Simplified Version of Data Management It s a collection of best data management practices that orchestrates Business and IT to work together in order to ensure the uniformity, accuracy, stewardship, consistency and accountability of the enterprise s core data assets. Source: Siili Solutions: Information Management Services
First, how to organize oversight organization Data Governance Oversight Policies and Standards Processes and Best Practices Requirements and Change Mgmt Issue Resolution Data Quality Metrics and Monitoring
Data Governance Oversight Organization Corporate level Steering Group N.N. Executive Sponsor Data Owners N.N. Customer Data Owner N.N. Product Data Owner N.N. Contract Data Owner Data Council Group Stakeholders BI ICT Services Data Champion Data Quality Manager Stakeholders Business Area A Business Area B Legal Business Area C Data Stewardship Group
Data Governance Oversight Organization Business activities are guided with organizational structures Functional organization Project organization Etc Why a new organization on top of all existing organizations?!
Data Governance Oversight Organization Because same data is shared across all business units, process and projects! Data definitions and standards have to be agreed across all of these other organizations. Data management must cross-over all alternative organizational boundaries. Data is defined according to data domains. E.g. clinical data, customer data, financial data, address data. All units, processes, projecs etc should use same definitions and standards.
There is software tools for supporting Data Governance! For example, Collibra Data Governance Center And many major software vendors have their own applications integrated to their platforms
That was Oversight Organization What about Data Policies and Standards Data Governance Oversight Policies and Standards Processes and Best Practices Requirements and Change Mgmt Issue Resolution Data Quality Metrics and Monitoring
Data Standards
Data Standards Data standards are documented agreements on meanings, structures, representations, transformations, uses and management of data. Currently, there is no single data standard that could be chosen and applied to manage all data for all business purposes. Therefore, organizations have to integrate existing standards to make them fit for their own business purposes in local historical settings and evolving business environments.
Data Standards in practice Business Vocabulary Data Definitions Data Models Person is a natural human being, having legal rights and responsibilities. High level conceptual definitions based on business perspective. Valid person is a data instance with Finnish Person Identification. Different levels of definitions based on logical and physical data elements. Persons can have relationships to other people and they can own products. Structures and rules linking individual data elements. The problem is that often these three layers are fragmented across software systems and business units.
Are all Information these data Production models syncronized Process (IPP) precisely consists across of three all data? phases based on Total Quality Management DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Human Perspective Enters data for primary purpose Builds data sets for other uses Analyses and reports data Interprets data and makes decisions for secondary purposes Technical Perspective Data Models DATA SOURCE A Data Definition Data Models DATA SOURCE B Data Models DATA SOURCE C Processing Block Data Definition Processing Block Processing Block Data Models STORAGE A Data Definition Processing Block Data Definition Data Models STORAGE B Data Definition INFORMATION PRODUCT A INFORMATION PRODUCT B INFORMATION PRODUCT C INFORMATION PRODUCT D Data Models Data Definition Data Models Data Models Data Models Data Definition
Often, Information they are not Production actually syncronized Process (IPP) and consists meaning of can three change! phases based on Total Quality Management DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Human Perspective Will not be updated if patient stays overnight! Does not copy codes manually from system to system! Does not mark all visits as ambulatory! Builds data sets for other uses Analyses and reports data Interprets data and makes decisions for secondary purposes Electronic Patient Record System Negative Processing Block error rate Billed Statistical from Data Warehouse patient as AP Negative Processing Block error rate Administrative Reports Ambulatory Procedures = 10726 Technical Perspective Radiology System No Not Integration! used Operation Room System Positive Processing Block error rate Planned as AP OLAP Cube Operation Room Reports Ambulatory Procedures = 15687
Standardization should reveal all meanings not force a single meaning! Data standards should document enterprisewide agreements on all semiotic levels syntactic: the structural properties; semantic: the meaning; pragmatic: the use and practices At syntactic level, there exists syntax standards that can be applied to every piece of data. For example, e-mail address has a certain shape with a username, internet domain and an @ sign in the middle. Personal Identification Code has a certain length, format and rules to build it. Syntax rules can be automated and they should be implemented to software systems to prevent technically erroneous data. At semantic level, even seemingly simple data elements, such as emails, can be revealed to be a complex data. THE TIP OF THE DAY Standardization should not lead to ambiguous agreements or force administrative documentation that hinders business processes. In practice, standardization should lead to a shared recognition of all alternative meanings and to support of different business requirements! People might have different emails such as student, employee or customer emails from wide variety of service providers. Often, these alternative meanings are not documented precisely. There can exist only ambiguous emails or addresses across systems. At pragmatic level, data can be used for different purposes with varying levels of activity. For example, emails can be based on studies, working, billing or private discussions.
That was data standards What about Data Quality? Data Governance Oversight Policies and Standards Processes and Best Practices Requirements and Change Mgmt Issue Resolution Data Quality Metrics and Monitoring
Data Quality
The definition of Data Quality Management Data Quality Management includes all the tools and processes that result in the creation of correct, complete and valid data that enables reliable and data driven decision-making. To ensure the reliability of company s core data elements, Data Quality should be measured the same way as companies measure e.g. their financial numbers or customer satisfaction through different key performance indicators.
Common data quality issues PEOPLE CAN T FIND DATA INCORRECT DATA POOR DATA DEFINITION PRIVACY & SECURITY INCONSISTENCY ACROSS SOURCES TOO MUCH DATA ORGANIZATIONAL CONFUSION 30% of time searching, unsuccessfully half the time 10-25% records contain inaccuracies Frequently misinterpreted, can t share between departments Subject to loss, risk of identity theft The norm with multiple processes, many duplicates Half never used, uncontrolled redundancy What data is important? How much is there?
Data standardization guides Data Quality Management Information Production Processes should be standardized, controlled, monitored and fixed in a systematic way: define: plan goals and processes; Define control: execute processes; React Control monitor: observe deviations between plans and execution; react: make adjustments to goals and processes Monitor
Every Information process and Production process step Process should (IPP) include consists quality of three controls! phases based on Total Quality Management DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Human Perspective Will not be updated if patient stays overnight! Does not copy codes manually from system to system! Does not mark all visits as ambulatory! Builds data sets for other uses Analyses and reports data Interprets data and makes decisions for secondary purposes Define Define Define Define Technical Perspective Electronic Patient React Record System Monitor Radiology System Define Control Negative Processing Block error rate React Monitor No Not Integration! used Define Control Billed Statistical from Data React Warehouse patient as AP Monitor Control Negative Processing Block error rate React Monitor Define Control Administrative Reports Ambulatory Procedures = 10726 Operation Room React System Control Positive Processing Block error rate React Control Planned React as AP OLAP Cube Control Operation Room Reports Ambulatory Procedures = 15687 Monitor Monitor Monitor
Data Quality is measured in relation to dimensions Data quality is usually measured in relation to data quality dimensions, such as accuracy, currency, and completeness. In practice, measurements must be developed locally to support local business needs.
Some of the most common data quality dimension and their meanings
Data quality is contextual and related to each individual use case For example, a list of email addresses might be enough accurate for sending mass marketing emails but the list might not be enough accurate for sending subscription bills for individual customers. In the first case, it is beneficial to increase the coverage of email campaign even though some of the emails might miss the targeted individuals. In the second case, it is critical to contact absolutely every single specified individual and only them to deliver their personal phone bills completely accurately. THE TIP OF THE DAY You need to develop and configure measurements for each dimension and use case individually! Data Quality management does not yet have mature out-of-the-box measurement systems.
Types of Accuracy Errors ACCURATE VALUES INACCURATE VALUES VALID VALUE AMBIGUOUS VALUE INVALID VALUE MISSING VALUE RIGHT VALUE RIGHT REPRESENTATION WRONG REPRESENTATION WRONG VALUE email@helsinki.fi email(at)helsinki.fi fake@boston.us email@aalto.fi 358231412432 "Correct email" "Correct email" "Fake email" "Student email" "wrong data" "No value" How to recognize? How to fix or prevent? How to fix or prevent? How to fix or prevent? How to fix or prevent? How to fix or prevent? Easy format change Verification Constraints Constraints Redefine
Data Quality Management is a complex combination of human and technological issues! That was just one dimension accuracy! How many other ways data can go wrong in relation all other dimensions?
Data Quality Management is a critical business issue! Internationally it has been estimated, that data quality costs 8-12% of revenue in typical enterprises while in service companies losses can be even 40-60% of total expenses. Data quality errors cause unnecessary rework, invalid decisions and dissatisfaction: Checking the data multiple times to fix errors Repeating work tasks that failed for using wrong data Financial losses because of invalid decision based on inaccurate data Inability to act due to lack of trusted information Customer and employee dissatisfaction due to problems in services
There is software tools for Data Quality! For example, DataCleaner Alternatively, Ataccama And many more
Summary
Data Governance from the perspective of football-players Positions Roles&Responsibilities Tactics Processes Rules Policies&Standards 4-4-2 formation Long-ball/Direct football Field surface Matches may be played on natural or artificial surfaces, according to the rules of the competition. The colour of artificial surfaces must be green. Football Bible
Data Governance from the perspective of football-players DATA GOVERNANCE Roles&Responsibilities DATA QUALITY Tactics&Processes DATA STANDARDS Policies&Standards Oversight Organization Data Processing Quality Controls Data Definitions Data Models Football Bible
Data Governance is needed to manage complexity and to enable Valid Decisions from Information Products Roles&Responsibilities Processes&Practices Policies&Standards DATA SUPPLY DATA MANUFACTURING DATA CONSUMPTION Analyses and reports data Human Perspective Enters data for primary purpose Builds data sets for other uses Interprets data and makes decisions for secondary purposes DATA SOURCE A Processing Block INFORMATION PRODUCT A STORAGE A INFORMATION PRODUCT B Technical Perspective DATA SOURCE B Processing Block INFORMATION PRODUCT C DATA SOURCE C Processing Block Processing Block STORAGE B INFORMATION PRODUCT D
Data-Driven Organization must manage their data systematically Managed with Data Governance Fixed with Data Standardization SEMANTIC ERRORS How many ambulatory procedures? DATA ERRORS 88 124 Fixed with Data Quality Controls
Train to control the game - mistakes are inevitable Develop new tactical processes information can be produced in different ways Get the best players skills and expertise cannot be replaced You will all eventually deal with data management related issues, better to be prepared. Organize teams - It s teamwork Understand the rules make sure everyone understands them
QUESTIONS? Sami Laine Data Science Architect, Siili Solutions Oyj, Finland Aalto University, Department of Computer Science and Engineering, Finland sami.laine@siili.fi sami.k.laine@aalto.fi https://www.researchgate.net/profile/sami_laine/ https://www.linkedin.com/pub/sami-laine/2/a61/970
Suggested reading Good to know: Redman, T. C. (2008): Data Driven profiting from your most important business asset. Harvard Business School Press. Davenport & Harris (2007): Competing on Analytics the New Science of Winning, Harvard Business Review Press Sarsfield, S. (2009): The Data Governance Imperative A business strategy for corporate data. IT Governance Publishing Wang, R. Y., Lee, Y., Pipino, l., and Strong, D. (1998): Managing your information as a product. Sloan Management Review. Summer 1998, pp. 95 106. Deep understanding: DAMA International (2010): The DAMA Guide to the Data Management Body of Knowledge. Technics Publications. Loshin, D. (2009): Master Data Management. MA: Morgan Kauffman Publishers.