Smart IT Argus456 Dreamstime.com From Data to Decisions: A Value Chain for Big Data H. Gilbert Miller and Peter Mork, Noblis Healthcare, transportation, finance, energy and resource conservation, environmental sustainability, and homeland security are but a few of society s grand challenges that look to information systems for efficient management and, more importantly, quality outcomes and solutions. Regardless of the specific challenge, underlying technologies and evolving user requirements continue to expand both data volume and variety. Data is coming from every imaginable source, often in real time, and the stakeholder demand for quality outcomes has never been higher. Given this perfect storm of expected outcomes, enabling technologies, and demanding stakeholders, enterprises face the daunting task of blazing a path that goes from raw data to quality outcomes. This task is made more complex with larger, faster, and more varied data, which doesn t automatically translate into more useful information. With exponential growth in data, enterprises must act to make the most of the vast data landscape to thoughtfully apply multiple technologies, carefully select key data for specific investigations, and innovatively tailor large integrated datasets to support specific queries and analyses. All these actions will flow from a data value chain a framework to manage data holistically from capture to decision making and to support a variety of stakeholders and their technologies. Defining a Data Value Chain Over a decade ago, Michael E. Porter introduced the value-chain concept, describing it as a series of activities that create and build value. Eventually, these activities culminate in total value, which the organization then delivers to its customers. 1 Figure 1 shows a proposed data value chain, which aims to manage and coordinate data across the service continuum from data generators to information consumers seeking to make decisions; form a collaborative partnership and coordinate data collection from various stakeholders and analyze that data to optimize service delivery and quality decisions; streamline data management activities to enable positive outcomes for all relevant stakeholders; and establish a portfolio-management approach to invest in people, processes, and technology that maximize the value of the combined data and inform decisions that enhance the organization s performance. These goals are accomplished through data discovery, integration, and exploitation. Data Discovery Before an organization can perform the analyses needed to support informed decision-making, it needs to know what data resources are available. Discovery includes not only inventorying data assets but also preparing and organizing these assets. Collect and Annotate The first link in the chain involves creating an inventory of available data sources and the metadata that describes the quality of those sources in terms of completeness, validity, consistency, timeliness, and accuracy. The emphasis is on turning unstructured data into structured data associated with valid metadata. Two techniques are suitable for data collection and annotation. The first is the Dublin Core, which 1520-9202/13/$31.00 2013 IEEE Published by the IEEE Computer Society computer.org/itpro 57
Smart IT Data discovery Data integration Data exploitation Collect and annotate Create an inventory of data sources and the metadata that describe them. Prepare Enable access to sources and set up access-control rules. Organize Integrate Analyze Visualize Identify syntax, structure, and semantics for each data source. Establish a common data representation of the data. Maintain data provenance. Analyze integrated data. Present analytic results to a decision maker as an interactive application that supports exploration and refinement. Make decisions Determine what actions (if any) to take on the basis of the interpreted results. Figure 1. The data value chain. The chain provides a framework with which to examine how to bring disparate data together in an organized fashion and create valuable information that can inform decision making at the enterprise level. focuses on supplementing metadata vocabulary terms with existing methods to describe, search, and index Web-based metadata. The second is the Department of Defense Discovery Metadata Specification, which focuses both on the process of developing a central taxonomy for metadata and defining a way to discover resources using that taxonomy. Historically, data standards have been haphazardly adopted to organize, represent, and encode information, which has prevented information sharing and data reuse later in the data value chain. Prepare The next task is to establish access to the data sources by copying them into a shared system and setting up access-control rules that is, security and privacy restrictions for data use. Massively parallel distributed storage systems, such as Hadoop Distributed File System, Big Table, and MongoDB, enable the storage of terabytes or more of data, regardless of structure. Tools for providing data access include representational state transfer, application programming interfaces, Web Services Description Language, and Open Database Connectivity/Java Database Connectivity. The extensible Access Control Markup Language provides a mechanism for specifying security and privacy policies. Languages for access-control policies have been around for decades, and role-based access control is well understood. Defining these roles across enterprise boundaries remains a challenge. Attribute-based access-control policies are less understood, but relevant standards are emerging. Standards for expressing and enforcing privacy policies are lacking. General-purpose tools enforcing privacy policies don t exist, and commercial packages tend to be tailored to specific environments. Organize The data source developer makes deliberate organizational choices about the data s syntax, structure, and semantics and makes that information available either from schemata or from a metadata repository. Either mechanism can provide the basis for tracking the shared semantics needed to organize the data before integrating it. Metadata repositories are commercially available, and numerous generic metamodels exist, many of which rely on Extensible Markup Language Metadata Interchange (XMI). However, because of XMI s generality, each tool provides customized extensions, which can lead to vendor lock, problems sharing schemata among participants, and other tool-interoperability issues. Analysts often skip formal data organization because they re more focused on their own data needs than on considering how to share data. However, sharing knowledge about internal data organization can enable more seamless integration with data providers environments (upstream) and data consumers environments (downstream). Data Integration The properly organized data is then ready to be combined into a common representation that suits a particular analysis. Each integration effort constitutes mappings that define how the data sources relate to the common representation. Metadata repositories need to be able to track these mappings to facilitate future analyses. Regardless of the particular representation as a community website or formal repository, such as a data warehouse combining disparate data sources delivers new, undiscovered information. Analysts can discover novel relationships between stakeholders or patterns that can point to abuses, such as fraud. Integration can be either virtual, such as through a federated model, 58 IT Pro January/February 2013
or physical, such as through a data warehouse. Traditional data federation technologies and emerging Semantic Web technologies support the integration and querying of combined data resources. Relational databases are suitable for most kinds of tabular data, while the Semantic Web is more compatible with nontabular, nonnumeric data, defined by rich networked relationships. Combining the two technologies will give data analysts a comprehensive toolkit for dataset exploration and for discovering the knowledge within integrated datasets. Data Exploitation Once the data has been gathered and integrated, an organization is ready to exploit it to make informed decisions. Decision makers rely on a combination of analyses that tease information out of the underlying data visualizations that convey those insights to the human. Analyze Integrated data sources are then ready for analysis, which includes maintaining the provenance between the input and results and maintaining metadata so that another analyst can recreate those results and strengthen their validity. Popular data analysis techniques, such as MapReduce, enable the creation of a programming model and associated implementations for processing and generating large datasets. This link, at the heart of the value chain, is perhaps the most mature in terms of available tools and techniques. Given this crowded marketplace, new offerings can more easily distinguish themselves not only on the basis of incremental analytic power, but also by providing strong integration with the links preceding and following analysis. By making it easier for analysts to access relevant data, for example, tool vendors provide differentiated value. These tools should also maintain the provenance among inputs and results so that other analysts can also understand and validate the results. Visualize Visualization involves presenting analytic results to decision makers as a static report or an interactive application that supports the exploration and refinement of results. The goal is to provide key stakeholders with meaningful information in a format that they can readily consume to make critical decisions. Industries, such as media and training, have a wealth of data visualization techniques, which others could adopt. Virtual and augmented realities, for example, enhance the user experience and make it easier to grasp information that s elusive in two-dimensional media. Although this technology has promising implications, virtual and augmented reality systems continue to be viewed as only suitable for training, education, and other highly customized uses. Make Decisions The final link of the data value chain is to determine what action is necessary given the visualized results. As supporting documentation, provenance information provides traceability to the original sources and their quality annotations, and the integration mappings and analysis metadata describe how analysts obtained the results. Key stakeholders can use the visualized results to change a negative behavior or reward a positive one. Understanding the underlying details of a particular problem and what contributes to that problem as well as what motivates the various constituents will inform stakeholders about required changes. Building on the existing analysis of data and incorporating additional data sources might reveal ways to more efficiently make decisions and take action. Data fragmentation is a significant obstacle to realizing value, and most data-driven enterprises don t give stakeholders any incentive to share their data. The proposed data value chain recognizes the relationship between stages, from raw data to decision making, and how these stages are interdependent. It s naïve to think that merely connecting data will reveal its wisdom. Low-quality data will not yield useful results, regardless of how clever the integration or query might be. Thus, the enterprise requires a plan that considers the entire continuum from the beginning of data collection to the final decisions based on that data. With more pressure to integrate data across the stakeholder continuum, the data value chain will support collaboration and data sharing and will provide structure and value even as the number and diversity of stakeholders grows. Furthermore, as stakeholders realize the benefits of data sharing, the entire enterprise and all stakeholders should begin to see improved operational quality and reduced costs. Reference 1. M.E. Porter, Competitive Strategy: Techniques for Analyzing Industries and Competitors, Free Press, 1998. H. Gilbert Miller is a member of IT Professional s advisory board and is corporate vice president and chief technology officer at Noblis. Contact him at hgmiller@ noblis.org. Peter Mork is a principal at Noblis. His experience includes information management, especially data integration, data discovery, data architecture, and information privacy. Contact him at peter.mork@noblis.org. computer.org/itpro 59