DELIVERING ON THE PROMISE OF BIG DATA AND THE CLOUD

Transcription

1 DELIVERING ON THE PROMISE OF BIG DATA AND THE CLOUD by Mark Jacobsohn Senior Vice President Booz Allen Hamilton Joshua Sullivan, PhD Vice President Booz Allen Hamilton WHY CAN T WE SEEM TO DO MORE WITH BIG DATA? We are living in an age inundated with information. Our world is increasingly instrumented sensors are collecting data on everything from hospital patients vital signs, to the moment-by-moment navigation of commercial aircraft, to consumer behavior based on buying patterns and the use of membership cards. Waves of data are coming from social media sites, from radio-frequency tracking systems, from the use of UPC barcodes. Our modern society is wired for data. Yet there is a growing belief in both business and government that we should be doing far more to take advantage of this wealth of information. We might use certain types of information for one purpose or another, but we nearly always view big data through multiple stovepipes, rather than treating it holistically. We do not appear to be able to tap the full potential of all the data available to us. We have the technical ability. There have been significant innovations in computer technology in recent years, particularly with the advent of cloud computing. Yet like the promise of big data, the promise of the cloud including unprecedented savings, much greater access to data, and better decision-making still seems largely unfulfilled. What holds us back is not technology, but a mindset. We are locked into an outmoded approach to data, one that relies on techniques created well before big data arrived on the scene. Those techniques give us access to only limited slices of information, and are not designed to easily connect an analyst with multiple sources of data. They were sufficient in their day, but are no longer enough. Ultimately, we are not doing more with big data because we do not have complete access to it. We are never able to use it all at once, and so we are unable to track overall trends, or see entire patterns, or ask complex questions that consider everything we know. To meet this need and take full advantage of both big data and cloud computing a new approach has been invented. Known as the Cloud Analytics Reference Architecture, it is the result of an ongoing collaboration between Booz Allen Hamilton and the U.S. government to leverage big data to search for terrorists and other threats. Intelligence analysts are now using the Cloud Analytics Reference Architecture to paint a comprehensive picture that incorporates the full range of intelligence data at once, including reports that have been amassed and are ongoing from the field. Unlike conventional techniques, this new approach makes it possible for analysts to use all available intelligence data, applying 2012 Booz Allen Hamilton Inc. All rights reserved. No part of this document may be reproduced without prior written permission of Booz Allen Hamilton. 1

2 an expanding set of analytic services to help them gain critical mission insights. The Cloud Analytics Reference Architecture, which is being adapted to the larger business and government communities, removes the traditional constraints by bringing together innovations in two areas of current technology. First, it uses the power of the cloud to put an organization s entire storehouse of data into a common pool, or data lake, making all of it easily accessible for the first time. It then uses sophisticated computer analytics, such as machine learning and natural language processing, to help extract the kind of knowledge and insight that creates value, guides strategy, and drives business and mission success. Although the Cloud Analytics Reference Architecture builds upon current techniques, it is not an incremental step forward. It is an entirely new approach one specifically designed for our new age of data. One way to understand how the Reference Architecture works is to view it in layers (see Figure 1). Its foundation is the cloud computing and network infrastructure, which supports the methods by which data is managed most notably, the data lake. The data lake, in turn, supports a two-step process to analyze the data. In the first step, special tools known as pre-analytics filter information from the data lake, and give it an underlying organization. That sets the stage for computer analytics in the next layer up to search for valuable knowledge. These elements support the final phase, the visualization and interaction, where the human insights and action take place. THE POWER OF THE CLOUD ANALYTICS REFERENCE ARCHITECTURE The Reference Architecture opens up the enormous potential of big data by allowing us to search for insight in new ways. It enables us to look for overarching patterns, and ask intuitive questions of all the data, rather than limiting us to narrowly defined queries within data sets. The Reference Architecture allows computers to take over much of the work humans are doing now freeing people to focus on the search for insight. It makes it possible for non-computer experts, for the first time, to frame the questions, look for patterns, and follow hunches. This is not some kind of magical solution far from it. The Reference Architecture is simply a new way of looking at data, but one that revolutionizes our ability to gain knowledge and insight. With conventional techniques, the data and analytics are locked into stovepipes, or silos. We can explore only limited amounts of data at any one time and then only with predetermined questions that have already been built in. The Reference Architecture removes these constraints by eliminating the silos, and consolidating all the information in the data lake. What results is not chaotic or overwhelming. Rather, the rich diversity of information in the data lake Figure 1. Primary Elements of the Cloud Analytics Reference Architecture 2 NOVEMBER 2012

3 becomes a powerful force. The data lake is more than a means of storage it is a medium expressly designed to foster connections in data. And the Reference Architecture explores those connections to search for valuable correlations and patterns This actually reduces the complexity of big data, making it manageable and useful, and creating efficiencies. Instead of using data to ask canned questions that test what we may already know, the Reference Architecture uses data to discover new possibilities solutions and answers that we have not even considered. The power of the Reference Architecture is that it constantly evolves and adapts as we search for insight, taking us beyond the limits of our imagination. WHAT THE CLOUD ANALYTICS REFERENCE ARCHITECTURE DOES The Cloud Analytics Reference Architecture removes the constraints created by data silos. While the rigid structures used in conventional techniques provide ease of storage, they carry severe disadvantages. They give us an artificial view of the world based on data models, rather than on reality and meaning. It is akin to reading a map through a tube we can never immerse ourselves in the diversity of big data, and instead make decisions based on limited and constrained information. Much of data science in the last ten years has been devoted to improving access to the silos and building bridges between them. But that does not solve the underlying problem that the data is regimented and locked in. Eliminating the need for silos gives us access to all the data at once including data from multiple outside sources. Users no longer need to move from database to database, pulling out specific information. And, because there are no data silos, there is no need to build complex bridges between them. If we want to know, for example, which parts of our computer network are most vulnerable to attack in the next six hours, we can take into account a wide variety of data sources at the same time. We might look at whether today is a holiday in certain foreign countries, which means that the young hackers known as script kiddies are more likely to be out of school and so have time on their hands to launch an attack. If we determine that a particular group is targeting us, we might examine how its members are connected, asking whether they had a common professor at a university, and if so, what techniques did he or she teach. The Reference Architecture gives us the ability to ask a full suite of questions rather than a pre-selected few. The Cloud Analytics Reference Architecture allows us to experiment more with the data. The Reference Architecture s flexibility provides a new kind of freedom to follow hunches wherever they may lead, to quickly shift direction to pursue promising avenues of inquiry, to easily factor in new knowledge and insights as they arise. With the conventional approach, it is difficult to add or switch variables that are not already part of a dataset or data base. That typically requires tearing apart and rebuilding both the structure that the data is in and the computer analytics that are custom-designed to handle specific lines of inquiry. The process is expensive and time consuming, and so consequently, we tend to focus instead on doing better analysis with the limited tools available on our narrow slices of data. With the Reference Architecture, we might decide, in the network security example above, to add new variables to the mix, such as the current propagation speed of commonly used viruses and botnets. Even if those variables come from outside data sources, we do not have to tear down and rebuild our data structures and analytics to consider them they seamlessly become part of our inquiry. The Cloud Analytics Reference Architecture allows us to ask more intuitive questions. With the conventional approach, we do not really ask questions of the data we create hypotheses, and then test the data to see whether we are right. In order to pose these hypotheses, we have to guess in advance what the answers might be, often a difficult proposition. To determine where our network is most vulnerable, for example, we would need to start with a hypothesis say, that any attacks will occur through outdated operating systems. That hypothesis, accurate or not, would drive our initial line of inquiry. With the conventional approach, we also need to be familiar with the data we are considering, including where it is (in what specific datasets or databases), what format it is in, and even to a large extent what the data itself contains. That level of knowledge might be achievable when we are working with a limited number of datasets or databases, but not with the vast amounts of information now becoming available to us. We often have to put aside, or assume away, factors that we might actually believe are critical. Add to these handicaps our inability to go beyond the pre-selected questions or easily change variables, and it becomes an impossible task. And so we never try it. We end up settling for marginal questions, and marginal answers. NOVEMBER

4 With the Reference Architecture, however, we can structure an inquiry around a single, intuitive, big-picture question: What part of our computer network is most vulnerable to attack in the next six hours? We do not need to know much about any of the data sources we are consulting the data will point us to the answer. The Cloud Analytics Reference Architecture allows us to more readily look for unexpected patterns it lets the data talk to us, so to speak. Even if we could ask all the questions we want, the way we want, there is simply too much data to formulate every question that might be important. Our questions can also be limited by our biases about the issues we are researching. We may not know what areas to explore, or what we should be looking at. To get the full picture, and help guide our inquiries, we need to see what patterns naturally emerge in the data. While we can look for patterns with the conventional approach, there are two significant drawbacks. We can only do such searches within our narrowly defined datasets and databases, rather than with the entire range of data available to us. We also must first guess what those specific patterns might be, and then test them out with hypotheses. But what about the patterns we do not even know might exist? How do we get to the hidden knowledge that often proves so valuable? Because there are no limiting data and analytic structures in the Reference Architecture, we do not need to pose hypotheses, and our search for patterns encompasses the entire range of data. For example, the U.S. military is now using the Reference Architecture to search for patterns in war zone intelligence data, to map out convoy routes least likely to encounter improvised explosive devices (IEDs). The Cloud Analytics Reference Architecture allows computers to take over much of the work humans are doing now enabling people to focus on creating value. Conventional methods require that people play a large role in processing the data including selecting samples to be analyzed, creating data structures, posing hypotheses, and sifting through and refining results. That intense level of effort may be workable for small amounts of data, but no organization has the personnel or resources to use that method to process big data. The Cloud Analytics Reference Architecture solves this problem by giving a great deal of that work to the computers, particularly tasks that are repetitive and computationally intensive. This reduces human error, and substantially speeds up the work. When we use the Reference Architecture to pose more intuitive questions, or to find patterns, we are essentially asking the computer to take us as close as it can to finding the answers we want. It is then up to us, using our cognitive skills, to find meaning in those answers. By separating out what the computer can do the analytics and what only people can do the actual analysis the Cloud Analytics Reference Architecture greatly eases the human workload. It is a division of labor that frees subject-matter experts to look at the larger picture. At the same time, the Reference Architecture rapidly highlights areas that analysts should not waste their time exploring enabling them to focus their time and attention in the right direction. For example, agencies that investigate consumer complaints against financial institutions often do not know which individual complaints are indicative of a broader patterns of consumer abuse, and so deserve the most attention. Investigators rarely have the time to sort through the vast array of sources that might provide valuable clues, such as blogs and social media sites where consumers commonly air their grievances. With a data lake that included all such available information, the Reference Architecture s analytics could quickly identify patterns, such as consumer abuse affecting large numbers of people. Investigators could then focus their resources on the most serious cases. The Cloud Analytics Reference Architecture s analysis capability enables subject matter experts to explore the data. If we are to drive business and mission success, we must give direct access to the data to the analysts, or subject matter experts, who understand what that success might mean. However, because of the high level of computer expertise needed to design custom data storage structures and analytics, much of the analysis today is conducted by computer scientists, computer engineers, and mathematicians acting as agents for the subject matter experts. They are typically the ones who translate the overall goals of the business and government analysts into the language of the machine. Whenever there is a middleman in any field, things tend to get lost in the translation, and data analysis is no exception. Here, it leads to a disconnect between the people who need knowledge and insight (the subject matter experts) and the data itself. It also substantially slows the process. In the top layers of the Reference Architecture, the middleman syndrome goes away. The ability to ask intuitive questions, and to look for patterns, provides the analysts with direct access to the data. That gives them the flexibility they need to experiment and explore, and allows the system to reach maximum velocity. The computer scientists, computer engineers and mathematicians still play a key role, but now are no longer the ones who drive the inquiries into the data. 4 NOVEMBER 2012

5 For example, investigators who suspect fraud may be occurring are often hampered by the need to go through computer experts to query the data. Their request may be one of many, and by the time they get back the information they need to act, the criminals have often long since committed the fraud and disappeared. With the Reference Architecture, however, investigators could query the data themselves, quickly pinpoint the fraud, and take action in time to stop the activity. THE FOUNDATION OF THE REFERENCE ARCHITECTURE: A NEW APPROACH TO INFRASTRUCTURE The Reference Architecture takes advantage of the immense storage ability of the cloud, though in a different way than in the past. With the conventional approach, cloud storage does not eliminate the data silos it simply makes them fatter. Organizations must continually reinvest in infrastructure as analytic needs change. Building bridges between silos, for example, typically requires reconfiguring and even expanding the infrastructure. The Reference Architecture, by contrast, has an inherent flexibility that enables organizations to pursue new analytical approaches with few if any changes to the underlying infrastructure. One reason is that the data lake is easily expandable. Because it stores information so efficiently, it can accommodate both the natural growth of an organization s data, as well as the addition of data from multiple outside sources. At the same time, the Reference Architecture replaces the current, custom-built analytics with a new generation of tools that are highly reusable for almost any number of inquiries. With the Reference Architecture, organizations do not need to rebuild infrastructure as their levels of data and analytics increase. An organization s initial investment in infrastructure is therefore both enduring and cost-effective. HOW THE DATA LAKE WORKS With the conventional approach, the computer is able to locate the information it needs because it knows precisely where it is in one database or another. The information is identified largely by its location. With the data lake, information is still identified for use, but now in a way other than by location. Specific pieces of information are identified by tags details that have been embedded in them for sorting and identification. For example, an investor s portfolio balance (the data) is generally stored with identifying information such as the name of the investor, the account number, one or more dates, the location of the account, the types of investments, the country the investor lives in, and so on. This metadata is what gets tagged, and is located by the computer during inquiries. The process of tagging information is not new it is commonly done within specific datasets and databases. What is new is using the technique to eliminate the need for datasets and databases altogether. The tags themselves are also a way of gaining knowledge from the data. In the example above, they might allow us to look for, say, connections between investors countries and their types of investments. The basic data the portfolio balance might not even be part of the inquiry. Such connections can be made with the conventional approach, but only if the custom-built databases and computer analytics have already been designed to take them into consideration. With the data lake, all the data, metadata and identifying tags are available for any inquiry or search for patterns. And, such inquiries or searches can pivot off of any one of those pieces of information. This greatly expands the usability of the data available to an organization. It actually makes big data even bigger. An important advantage of the data lake is that there is no need to build, tear down, and rebuild rigid data structures. For example, suppose we develop an improved approach to translating English into Chinese. With conventional techniques, the database is the translation. To make major changes, we would have to go back to the original data (the English and Chinese words), and build a completely new structure. With the Reference Architecture, however, we would simply pull out the data in a new way, easily reusing it. In addition, the data lake smoothly accepts every type of data, including unstructured data information that has not been organized for inclusion in a data base. An example might be the doctors and nurses notes that accompany a patient s electronic health records. Two other critical emerging data types are batch and streaming. Batch data is typically collected on an automated basis and then delivered for analysis en masse for example, the utility meter readings from homes. Streaming data is information from a continuous feed, such as video surveillance. Most of the flood of big data is unstructured, batch and streaming, and so it is essential that organizations have the ability to make full use of all types. With the data lake, there is no second-class or third-class data. All of it, including structured, unstructured, batch and streaming, is equally ingested into the data lake, and available for every inquiry. NOVEMBER

6 It is an environment that is not random and chaotic, but rather is purposeful. The data lake is like a viscous medium that holds the data in place, and at the same time fosters connections. Because the data is all in one place, it is, in a sense, all connected. GATHERING INFORMATION FROM THE DATA LAKE: THE PRE-ANALYTICS In the first step in analyzing the data, the Reference Architecture uses tools known as pre-analytics to filter data from the data lake and then give it an underlying organization. For example, a recent study by Booz Allen and a large hospital chain in the Midwest analyzed the electronic medical records of hundreds of patients, to track the progression of a life-threatening condition known as severe sepsis. Pre-analytics were used to first pull patients vital signs from a version of a data lake, and using the time-and-date stamps embedded in the records organize them in chronological order. Once that was accomplished, computer analytics could then search for patterns in the way the patients vital signs changed over time. Pre-analytics accomplish a number of tasks at once. Using the tags, they locate and pull out the relevant data from the data lake. They then prepare that data for the analytics, sorting and organizing the information in any number of ways. The pre-analytics allow great flexibility in the inquiries for example, one such tool might transliterate a name like Muhammad into every possible spelling (e.g., Mohammad, Mahamed, Muhamet). This would enable the computer to collect and analyze information about a particular person, even if that person s name is spelled differently in different sources of data. Although pre-analytical tools are commonly used in the conventional approach, they are typically part of the rigid structure that must be torn down and rebuilt as inquiries change. Generally, they cannot be reused for example, each name to be transliterated would require an entirely new pre-analytic. Because such work is resource-intensive, only a limited number of such tools can be built, severely hampering an organization s ability to make full use of its data. By contrast, the preanalytics in the Cloud Analytics Reference Architecture are designed for use with the data lake, and so are not part of a custom-built structure. They are both flexible and reusable, giving organizations almost endless windows into their data. Moreover, they are designed to be interoperable from the moment they come on-line, creating a set of easily shared services for all users of the data. THE POWER OF COMPUTER ANALYTICS Once the data has been prepared, the search for knowledge and insight can begin. As with the other elements of the Reference Architecture, computer analytics are used in an entirely new way. An analogy might be the difference between the smartphones of today and the separate functions for telephones, personal digital assistants and computers of the not-so-distant past. Smartphones do more than just combine those functions they create a new world of possibilities. The computer analytics in the Cloud Analytics Reference Architecture do the same. There are several types of analytics in the Reference Architecture, including: Ad hoc queries. These are the analytics that ask questions of the data. While in the conventional approach the analytics are part of the narrow, custombuilt structure, here they are free to pursue any line of inquiry. For example, a financial institution might want to know which of its foreign investors are at greatest risk of switching to another firm, based on dozens of characteristics of current and former customers. Later, analysts might want to change the question somewhat, asking the extent to which the political turmoil in certain countries plays a role. They can use the same analytic to ask the second question, and any number of other questions like the pre-analytics, they are flexible and reusable. And they enable the kinds of improvised, intuitive questions that can yield particularly valuable results. Machine learning. This is the search for patterns. Because all of the data is available at once, and because there is no need to hypothesize in advance what patterns might exist, these analytics can look for patterns that emerge anywhere across the data. Alerting. This type analytic sends an alert when something unexpected appears in the patterns. Such anomalies are often clues to the kind of hidden knowledge that can provide business with a competitive advantage, and help government organizations achieve their missions. Pre-Computation. These analytics enable organizations to do much of the analyzing in advance, creating efficiencies. For example, an auto insurance company might pre-compute the policy price for every individual vehicle in the U.S., so that, with a few additional details, a potential customer can be given an instant quote. 6 NOVEMBER 2012

7 PUTTING IT ALL TOGETHER: VISUALIZATION AND INTERACTION Decision-makers may be understandably concerned that all this big data will be overwhelming, that removing the tube from the map will simply lead to information overload. Quite the opposite is true. The Cloud Analytics Reference Architecture addresses the issue head-on by incorporating the visualization how the knowledge is presented to us into the analytics from the outset. That is, the analytics not only conduct the inquiries, they help contextualize and focus the results. At the visualization and interaction level of Reference Architecture, this focus enables the analysts to more easily make sense of the information, to frame better, more intuitive inquiries, and to gain deeper insights. Building the visualization into the analytics has another advantage it provides the ability for quick and effective feedback between the two layers, so that the presentation of the findings can be continually refined for the decision-maker. With the Reference Architecture, the flood of information is not overwhelming it is readied for action as never before. This breakthrough in visualization could have as profound an effect on decision-making as bar graphs and pie charts did in the 1950s and 1960s, when statistics became widely used in business. Those visuals presented all the essential information at a glance, changing the nature of decision-making. The Reference Architecture will do the same but this time with big data. DELIVERING ON THE PROMISE The possibilities of big data and the cloud are not pipe dreams. But they will not be fulfilled on their own conscious effort and deliberate planning are needed. Unless organizations make the right infrastructure decisions, they cannot hope to build a data lake. Unless they make the right data management decisions, they will never break free from the rigid data and analytic structures that are so limiting. The Cloud Analytics Reference Architecture can be seen as a road map for that decision-making, one that shows the importance of a holistic, rather than piecemeal, haphazard approach. Each element is closely tied to each of the other elements, and so all must be considered together. The Cloud Analytics Reference Architecture is no more expensive to build than traditional approach, and is considerably more cost-effective in the long run. Because the elements of the Cloud Analytics Reference Architecture are largely reusable, they can scale an organization s big data in an affordable way. The Cloud Analytics Reference Architecture is already being used by the U.S. government to make our nation safer, and it can help other organizations in government and business create value, solve real-world problems, and drive success. The grand promise of big data and the cloud is now within reach. FOR MORE INFORMATION Mark Jacobsohn jacobsohn_mark@bah.com Joshua Sullivan, PhD sullivan_joshua@bah.com This document is part of a collection of papers developed by Booz Allen Hamilton to introduce new concepts and ideas spanning cloud solutions, challenges, and opportunities across government and business. For media inquiries or more information on reproducing this document, please contact: James Fisher Senior Manager, Media Relations, , fisher_james_w@bah.com Carrie Lake Manager, Media Relations, , lake_carrie@bah.com NOVEMBER