INFORM: 3 MYTHS & 5 BIG DATA PROBLEMS 3 BIG DATA MYTHS THAT ARE HOLDING US BACK

Transcription

1 INFORMATION DRIVES SOUND ANALYSIS, INSIGHT, AND ACTION. DO YOU HAVE A BIG DATA PROBLEM? INFORM: 3 MYTHS & 5 BIG DATA PROBLEMS These days, as organizations refine their technology roadmaps and plan their future data infrastructures, there s often a lot of head scratching when it comes to the subject of Big Data. We ve all heard the hype. If our organizations don t embrace Big Data, we re going to get disrupted and become market dinosaurs. We must overhaul our toolsets and hire an army of data scientists, who are rumored to return more real business value than entire departments of product managers and marketing analysts. Predictably, the loudest voices declaring these new truths belong to vendors selling Big Data technologies. But just because Big Data has been overhyped doesn t mean that it is all hype. Plenty of organizations have successfully demonstrated strong returns in some cases, phenomenal returns from their Big Data initiatives. The potential these new tools and approaches offer is something that today s IT organizations must take seriously. Just as our organizations evolved their IT infrastructures step-by-step to become more service-oriented, so, too, will we steadily and deliberately put new Big Data foundations in place. To do so successfully and to cut through the marketers fog and noise we must first take a step back and ask, Do we have a Big Data problem? If the answer to that question is yes, then we next need to ask, What particular Big Data problem (or problems) do we have? The answers to these questions are key to determining how we should proceed on our journey into the realm of Big Data. 3 BIG DATA MYTHS THAT ARE HOLDING US BACK When IT teams begin evaluating Big Data technologies and their potential uses, three common myths tend to make the fog of hype even more difficult to cut through. MYTH #1: BIG DATA TOOLS ARE HIGHLY ADVANCED AND IMPOSSIBLE TO UNDERSTAND One of the hardest parts about getting comfortable with Big Data tools and techniques is simply understanding what each one does in the first place. Bombarded by streams of semi-academic jargon and acronyms, it can be quite difficult even for very technical people to get their hands around the basics of what a particular tool does. Conceptually, though, most Big Data tools aren t really all that hard to understand, provided you can find someone who can explain in clear language what each tool is meant to do, the basic way that it stores data or divvies up jobs for processing, and how that differs from the way traditional data management tools do things.

2 MYTH #2: BIG DATA IS A SINGLE KIND OF TECHNOLOGY Just as Business Intelligence is an umbrella term that sweeps in a broad array of tools, techniques, and ways of thinking about problems, so too is Big Data. Before we begin diving into the inner workings of specific technologies (like Hadoop or NoSQL databases or visualization tools) it helps to first understand the analytic purpose of each. In other words, we need to understand the data problems we are trying to solve before we can begin to evaluate and select the technology needed for the solution. MYTH #3: BIG DATA TECHNOLOGIES ARE INHERENTLY SUPERIOR TO TRADITIONAL RELATIONAL DATABASES AND BI TOOLS Too often new Big Data technologies are presented as fundamentally better than older data tools, and the decision to adopt them is posed as an either/or question. Do we stick with the boring, old traditional data stack we ve got or replace it with these new, shiny Big Data things? But third normal form relational databases management systems (RDBMSs) like Oracle, SQL Server, and MySQL do what they are designed to do very well, as do business intelligence (BI) tools and approaches like star-schema data models, OLAP cubes, and pivot tables. The new generation of Big Data technologies were not developed to do everything better than transactional RDBMSs or BI tool sets but rather to solve specific problems that the previous generation of tools were not designed to solve. 5 BIG DATA PROBLEMS When you get right down to it, there is a relatively small set of problems that Big Data techniques and technologies can help us solve. Here are five of them. Which of these problems is your organization facing today? The answer to that is the key to laying the future roadmap for your Big Data strategy. 1. UNSTRUCTURED DATA Unstructured data doesn t fit nicely into the rows and columns of relational databases or the facts and dimensions of OLAP cubes. Companies like Google figured this out pretty quickly when they started working to parse, store, and index a particularly unruly set of unstructured data: the raw HTML content of millions of web pages. The tools and techniques they developed can be used to manage many other types of unstructured data as well: the thousands of customer s a company receives each day, social media tweets and posts, information captured in contracts and policies in Word document or PDF formats. Traditional extract, transform, and load (ETL) tools and RDBMS schemas are typically not very good at parsing and storing such unstructured information. If you have terabytes of valuable business data locked away in unstructured documents and files, you have a Big Data problem.

3 2. LARGE VOLUMES OF DATA THAT ARE DIFFICULT TO PROCESS Enterprise-strength relational databases can handle tables with millions of rows of data just fine, but when those numbers climb into the billions things can start to grind to a halt. There are tricks and techniques that can be used to keep scaling up, but at a certain point you begin to hit the limits of what the technologies were designed to do. These days, though, we are asking our teams to manage an ever-increasing volume of data. Clickstream data from web sites, for instance, is typically nicely structured discrete columns of well-formatted data but a busy site can generate tens of billions of rows of it each year. Similarly, network-connected devices and instruments can spit out thousands of readings per second, and when you have thousands of machines or vehicles in your data landscape, that volume gets very big very quickly. Much of that data may not even be particularly important readings that indicate normal operations, for instance but hidden among them might be a few critical patterns that just once or twice a year indicate a problem that requires maintenance to avoid an expensive failure. As we move into the era of the Internet of Things, where everything from wearable fitness devices to smart refrigerators are now spitting out unprecedented amounts of data, this problem is only increasing. If loading up all that hay and looking for a few needles within it is over-extending your current data systems, you ve got a Big Data problem. 3. COMPLEX ANALYTIC TASKS THAT ARE DIFFICULT TO PROCESS Sometimes it s not the raw volume of the data that is challenging but the sheer amount of crunching we are trying to do on it. As the techniques of data science become more widely used in business, running complex correlations and regressions and statistical models within SQL scripts or statistical programming tools like SAS can start to take hours or days to run. They may also start to require their own sandbox environments to run in to keep from bogging down other reporting and analytic functions. Are your data analysts spending hours each day just extracting, downloading, and moving data between environments? Do they spend days on end coding and recoding their scripts to make simple tweaks and fix bugs? Are they sitting around sipping coffee in the break room while their latest analytic job runs? If so, you ve got a Big Data problem. 4. MESSY, INCONSISTENT DATA The more businesses try to tap into what their customers are thinking and saying to each other and to the world, the more they run into challenges of messy, inconsistent data. How can you predict, for instance, what hash tags and keywords your customers may dream up for the social media content they generate (#CustomerServiceFail)? How much of what they are telling your customer service representatives gets captured in free form notes?

4 Or, perhaps you just want to take customer data from a dozen or more internal systems, each of which stores a different subset of the data its own unique format. The typical pre-big Data approach would have been to create an enterprise-wide master data model, map all the various source data fields to it, and build a monumental set of ETL jobs to extract and shoehorn it into the master model. This works well if you have several years of project time and millions of budget dollars to accomplish it. If you don t have that time and money, you have a Big Data problem. 5. APPLICATION DATA THAT CHANGES FREQUENTLY One of the pains that organizations encounter as their data warehousing and analytics foundation matures is that, as the warehouse grows and number of source systems expands, small changes in the data elements in those source systems can have ever-bigger downstream ripples. Each application enhancement or release begins to create bigger and bigger warehouse, ETL, and reporting maintenance tasks, and the data management teams begin to spend an ever greater percentage of their time just keeping the existing data infrastructure running and not bringing new value to the business. A common response is to dig in and try to do more of the same thing faster, or to put on the brakes and slow down the pace of application data changes. Release cycles expand, and project costs balloon. Perhaps there are other ways to handle data source changes other than through the full ETL, warehousing and reporting pipeline? If you are asking yourself this question, you have a big data problem. TRANSFORM: GETTING STARTED IDENTIFYING YOUR BIG DATA PROBLEMS In many cases, a particular type of data may pose two or more of these sorts of Big Data problems. A Twitter stream, for instance, can have a large volume of data that s difficult to process as well as contain messy, inconsistent data. The Big Data tools you need to adopt will vary depending upon which Big Data problem (or problems) you have, and most organizations find that no single tool Hadoop, document-oriented NoSQL databases, columnar and sharednothing NoSQL databases will solve all their problems. Like a repairman with a wellstocked toolbox, they frequently end up with a portfolio of Big Data technologies that they use selectively depending upon the current problem on which they are working. Here are a few questions that can help us begin to understand the current data landscape within an organization and whether there s a problem that can be addressed by Big Data technology: 1. What data sources do we need to deal with today and in the future? 2. What are we going to do with that data? Who needs to analyze it, and what type of analysis do they need to do?

5 3. Can we achieve the objectives with our current data infrastructure? If not, what are the specific barriers or limitations preventing us from doing so? In other words, what specific Big Data problem or problems are posed by those this particular data source and our analytic objectives for it? 4. How much does completeness and accuracy matter? 5. How frequently and significantly is the format or structure of this data source likely to change over the next 12 to 24 months? Answering these questions will help you identify the Big Data problems your organization is facing, and from there you can begin evaluating the proper tools and techniques to solve that problem. You might even discover that many of the problems you are trying to solve aren t Big Data problems at all and that the proper way to approach them is by improving the way you use the tools you already have in-house or by putting in place structured data governance, data stewardship, and data evangelism approaches. When it comes to mapping out the future of your Big Data infrastructure, we can get a lot further a lot faster if we ignore the hype and the grand promises of technology promoters and focus instead on understanding the specific data problems we are trying to solve. ABOUT THE AUTHOR Robert Moss leads Optimity s Technology Platforms advisory practice. An experienced technology and strategy leader, he helps organizations understand and adapt to the new technologies that are disrupting traditional business models. Robert advises clients in a range of industries, including healthcare, media & entertainment, and insurance, and supports them as they formulate and execute technology strategies such as online commerce, mobility, and advanced data management. Prior to joining Optimity Advisors, Robert served as Vice President of Product Development and Vice President of Product Management for Benefitfocus, the largest provider of Softwareas-a-Service (SaaS) solutions to the commercial health insurance industry. At Benefitfocus, he was responsible for the strategic product direction for the company s online benefits platform, which is used by over 11 million consumers and 300,000 employers to shop for, enroll in, and pay for their health insurance and other benefits.

6 Washington, DC 1600 K Street, Suite 202 Washington, DC Phone: Fax: info@optimityadvisors.com New York, NY 183 Madison Avenue, Suite 1205 New York, NY Phone: newyork@optimityadvisors.com Optimity Advisors Robert Moss Robert.Moss@optimityadvisors.com 1600 K St. NW, Suite 202 Washington, DC Direct: Main: Fax: Los Angeles, CA 1100 Glendon Avenue, Suite 925 Los Angeles, CA Phone: losangeles@optimityadvisors.com London Office 1st Floor, Kemp House City Road London EC1V 2NP UK Phone: +44 (0) enquiries@matrixknowledge.com Brussels Office Square de Meeus 25 B-1000 Brussels Belgium enquiries@matrixknowledge.com Offices also in Sacramento, Minneapolis and Dallas