1
Sources: The Guardian, May 2010 IDC Digital Universe, 2010 IBM Institute for Business Value, 2009 IBM CIO Study 2010 TDWI: Next Generation Data Warehouse Platforms Q4 2009 Summary Data is exploding in volume, variety and velocity. And both struc and unstruc info will continue to grow at astronomical rates. This creates a tremendous opportunity for organizations to make timely decisions and achieve business goals. However, at the same time, organizations are struggling to gain deeper insights from this data. Business leaders continue to make decisions without access to the trusted information they need. CEOs understand that they need to do a better job in capturing and understanding information 3
Today organizations are only tapping in to a small fraction of the data that is available to them The challenge if figuring out how to analyze ALL the data, and find insights in these new and unconventional data types Imagine if you could analyze the 12B TB of tweets being created each day to figure out what people are saying about your products, figure out who the key influencers are within your target demographics. Can you imagine being able to mine this data to identify new market opportunities. What if hospitals could take the thousands of sensor readings collected every hour per patients in ICUs to identify subtle indications that the patient is becoming unwell, days earlier that is allowed by traditional techniques. Imagine if a green energy company could use PBs of weather data along with massive volumes of operational data to optimize asset location and utlization, making these environmentally friends energy sources more cost competitive with traditional sources. Imagine if you could make risk decisions, such as whether or not someone qualifies for a mortgage, in minutes, by analyzing many sources of data, including real-time transactional data, while the client is still on the phone or in the office. Image if law enforcement agencies could analyze audio and video feeds in real-time without human intervention to identify suspicious activity. As these new sources of data continue to grow in volume, variety and velocity, so too does the potential of this data to revolutionize the decision-making processes in every industry. 4
This is the Big Data opportunity. We define it as the opportunity to extract insight from an immense volume, variety and velocity of data, in context, beyond what was previously possible. Massive volume, variety and velocity are defining characteristics of Big Data. In order to capitalize on this opportunity, enterprises must be able to analyze ALL types of data relational and non relational. Texts, sensor data, audio, video, transactional. Sometimes, getting an edge over your competition can mean identifying a trend, problem or opportunity, seconds, or even microseconds before someone else. More and more of the data being produced today, has a very short half-life. Organizations must be able to analyze this data in real-time if they are to be able to find insights in this data. And, as implied by the term Big Data, organization are facing massive volumes of data. Organizations who don t know how to manage this data, are overwhelmed by it. But the opportunity is, with the right technology, to analyze ALL the data, to gain a better understanding or your business, your customers, the marketplace. 5
Most of you know of Watson, our computing system designed to compete on the Jeopardy game show. Watson represents a breakthrough in terms of volume of information stored, and the ability to access it quickly (answering natural language questions). Watson is impressive because there are many commercial uses for this technology and the technology exists today Watson answers a Grand Challenge: Can we design a computing system that rivals a human s ability to answer questions posed in natural language, interpreting meaning and context and retrieving, analyzing and understanding vast amounts of information in realtime? IBM Watson is a breakthrough in analytic innovation, proving that it is possible to harness vast amounts of information and rival a human s ability to answer questions posted in natural language in real-time. But it doesn t matter how good the machine is if we don t have good information to feed it. Imagine the possibilities. We live in a time where a computer can compete against humans at answering questions in plain English, based on storing, retrieving, analyzing and understanding vast amounts of information at real-time speeds. These same capabilities can enable you to improve and optimize your business, too. IBM just showed the value of putting that information to work by creating a computing system capable of competing on Jeopardy 6
Watson of course, leveraged Big Data technologies in order to conquer the grand challenge Watson used Hadoop for building it s knowledge base using it to distribute the workload for loading information into its corpus.... That was approximately 200 million pages of text. Hadoop is the same innovative technology that has been commercialized within InfoSphere BigInsights, of the key components of our Big Data platform. The work has already begun to commercialize the technology in Watson.. And we expect the relationship between Big Data and Watson to grow. Our Big Data platform will be the ideal vehicle for distilling Big Data into insights that can be used by Watson for advanced search and analysis. 7
Dr. Carolyn McGregor, Research Chair in Health Informatics at the University of Ontario Institute of Technology has been exploring new approaches for the last 12 years to provide specialists in neonatal intensive care units better ways to spot potentially fatal infections in premature babies. Changes in streams of real-time data such as respiration, heart rate and blood pressure are closely monitored in her work and now she is expanding her research to China. "Building upon our work in Canada and Australia, we will apply our research to premature babies in rural and remote hospitals in China. With this new additional data, we can compare the differences and similarities of diverse populations of premature babies across continents," said Dr. Carolyn McGregor of the University of Ontario Institute of Technology. "In comparing populations, we can set the rules to optimize the system to alert us when symptoms occur in real time, which is why having the streaming capability that the IBM platform offers is critical. The types of complexities that we're looking for in patient populations would not be accessible with traditional relational database or analytical approaches." 8
First one all kinds of data 3 points scale = volume, variety = 2, 5 = velocity Point it has to deal with all 3 Major point 1 handle the 3 Vs Major point 2 how it handles the #vs flexible, react to volatility Major 3 Ease of use for developers and users Major 4 Ent class Major 5 extensive integration 9
Key Points Big Data platform is built upon open source. We ve embraced open source movement because we believe the Hadoop technology is the correct one to address internet scale analytics. But our approach is to mature and build upon that technology for an enterprise class platform. Open source built on Hadoop (map reduce, HDFS), HBase (Hadoop database), Pig (analysis of large data sets, high level language for data analysis programs), Lucene (full text search), Jawl (query language for Javascript Object Notation) We ve matured it with two enterprise engines for processing large volumes of data and analyzing a variety of data. Streaming analytics is designed to manage stream flows and apply various analytics mining, mathematical, video, etc. against that streaming data. Internet scale analytics is designed to store data at rest, as-is, and apply analytics, such as text analytics against that data set. User Environments This is an important part of maturing the platform and exposing the power of big data to existing resources, not just specialist programmers who can write map reduce programs. The develop environment is designed to provide a mature environment for developing and testing Big Data analytics and applications. There are end-user visualization capabilities to explore the data and analyze it. Integration this is an important aspect of the BD platform it had to be integrated in order to bring big data to the enterprise the insight has to be integrated to warehouses, databases, applications, etc. One of the key vehicles for doing that is Information Integration which includes governing that data. Proof Points & Stats Tera Echos our developers can deliver apps 45% faster due to the agility of the streams processing language shows how a mature development environment and language speeds development of BD apps. 10
Key Points Let s take a look at one of those previous use cases in more detail the 360 degree multichannel customer sentiment analysis. The objective is to determine a customer s mood based on their interactions across many channels. First, there are many sources for this analysis website logs and data, social media, and call centre reports and logs. The website data and social media data is managed with the internet scale analytics portion of the BD platform bringing in huge volumes of data from social media sits like facebook or twitter. Call center data is monitored in real-time via streaming analytics. The Big Data platform can determine sentiment, and sentiment changes for example if a customer experiences poor service and complains via the call center, or blogs about it on facebook. That sentiment would be shared via information integration to the data warehouse for further analysis to answer questions such as what is the value of this customer? Do we care about their negative sentiment? What should we do? From there, the warehouse would trigger many potential actions further sentiment analysis and visualization with Cognos Consumer Insight, or a campaign with Unica, or to store that sentiment fact in MDM to trigger different actions in the operational systems integrated with MDM. Customer Stories Bharti uses streams to detect call centre channel sentiment and mediation. Retailed (Shop Direct can t use their name) uses BD platform for internet web commerce analysis. Most clients start with one channel and incorporate many over time. 11
12
Key Points Key point need to have analytics that are built for purpose essentially built for variety of data sources. Big Data platform cannot just be a platform for processing data, it has to be a platform for analyzing that data. IBM significant differentiator we have many purpose built analytics capabilities. Many of these have come out of research. Test analytics this is an important component of the BD platform, as much of the data, while semi-structured, has its real content inside the text of specific attributes (e.g., Facebook the value may be in an attribute called post the real meaning, the real structure is in the text itself therefore text analytics is a very key component to the BD platform. There are other key analytics video, acoustic, financial, mining, stats the real point is they are built for variety, and not found in any other vendor s BD strategy in the market. Customer Stories Terra Echoes Covert analysis system, Acoustic signals from buried fibre optic cables are monitored, analyzed, and reported in real-time. Designed to scale up to 1600 streams of raw minary data. 13
Key Points Trust, governance, privacy how you use data for the enterprise matters this isn t just a technology for an internet company, this is managing large volumes of potentially sensitive data for the enterprise. Govern what comes it, govern what goes out - How you use Big Data matters Even though Big data means all of the data, it doesn t necessarily mean you bring in all of the data and expose it to everyone without any sort of governance or quality. Example of internet tweets or blog posts on upcoming M&A, it could be factored into brand sentiment analysis, but what if you are not supposed to factor that data into internal decision making? 14
T r a n s c r i p t : a n d t h e n a p p l y i n g t o o l i n g t o i t t o m a T r a n s c r i p t : a n d t h e n a p p l y i n g t o o l i n g t o i t t o m a
24
26
Lower Total Cost of Ownership User interfaces bring big data to the everyday user Integration with enterprise technologies Less complex Platform = 1 stop shop The platform has brought together all the necessary components into one Lower risk Grow from POC to enterprise production Increase revenue by finding new insights Leading engines for complex analytics 28
29