Veracity of data. New approaches are emerging to account for uncertainty in data at a giant scale IBM Corporation

Transcription

1 Veracity of data 1. The degree to which data is accurate, reliable, certain 2. An emerging platform for organizing, understanding and deriving value from big data Introduction Financial decisions require confidence in the veracity of data on which they are based. The challenge is that by 2015, 80 percent of the world s data will be uncertain. Data uncertainty arises from many sources and in many ways: as inconsistencies, ambiguities, technical errors, time delays, assumptions, and even sarcasm and outright deception. The challenge is how to deal with uncertainty when analyzing big data. How do we represent uncertain data, reduce the uncertainty of data, and reason about that data in a way that allows us to make sound business decisions? New approaches are emerging to account for uncertainty in data at a giant scale. 1

2 Step one Establish the base Every organization has data it can trust. This base data comes from reliable sources and provides the foundation for making sense of less certain forms of data. A key step in managing uncertain data is establishing and providing a unified view of the base so that each department has access to a single, trusted source of data. 2

3 Step two Gather more data An instrumented and interconnected world offers an abundance of new data. Most of this new data is low veracity: it comes from uncertain sources, such as social media, blogs or video. When handled correctly, this data provides context and increases the value of our base data. Start by examining as much data as possible from a variety of sources. 3

4 Step three Combine the data Integrating related data helps maximize context and reduce uncertainty. For example, mobile location data in complex environments such as cities is notoriously imprecise. By combining data with recent social updates and accelerometer data and correlating that with weather data or past purchases CFOs can gain a better understanding of where our customers are and how likely they are to buy. That information, in turn, can be used to make better immediate-term financial forecasts and decisions. 4

5 Step four Extract insights Analyze and extract insights from all available data. Tools such as natural language processing, customer predictive analytics and social media sentiment each go beyond the basics to extract meaning, reveal insights and build context awareness. Be clear about the questions you want to answer so you can employ the right set of analytics for the job. A tweet about lobster means one thing if used to describe a food allergy and something entirely different when used to characterize a sunburn. Tools such as natural language processing (NLP) help to understand nuance, sarcasm and context. 5

6 Step five Build models and scale to the organization This includes building models that incorporate uncertain data. Mathematical approaches such as stochastic reasoning or Monte Carlo simulations take uncertainty into account to yield predictive recommendations. To foster a culture that values data-driven decisions, make these models understandable and accessible across the organization. A tweet about lobster means one thing if used to describe a food allergy and something entirely different when used to characterize a sunburn. Tools such as natural language processing (NLP) help to understand nuance, sarcasm and context. Automated diagnostic models, such as those used by IBM s Watson, take the uncertainty of data into account to provide physicians with a list of probable diagnoses alongside corresponding measures of likelihood. 6