1 Navigating the big data challenge Do you have lots of data but few insights? By Rasmus Wegener and Velu Sinha
2 Rasmus Wegener is a partner with Bain & Company in Atlanta. Velu Sinha is a partner in Bain s Palo Alto office. Both are members of Bain s Technology practice. Copyright 2012 Bain & Company, Inc. All rights reserved.
3 Navigating the big data challenge Twenty years ago, business leaders wanted more data to inform their decisions. Today they have so much data available through indiscriminate collection and easy access to millions of data sets that decision making has actually become more difficult. Significant insights don t spring from raw data. Someone has to know which questions to ask of the data and where to find the answers. Many companies face this problem, as they look to these vast stores of data to learn more about their markets, their customers and their opportunities. In addition to being creative, curious and business savvy, effective decision makers are learning to become more comfortable working with data and more capable of drawing insights from analysis. For executives who have long wished for better answers, big data can look like an easy win. They might be tempted to think that an expensive big data solution, on its own, can sort through data and deliver the new insights they are looking for. As with any technological innovation, the question of how to use it is ultimately a business question. No big data product or service can substitute for the rigorous and demanding process of figuring out what questions to ask. We find that when companies go through the work of determining the right questions to ask and where to find the answers, they typically find that the tools are already inhouse, either in business intelligence software or in existing database tools. To help executives decide whether they need to invest in new talent, tools and capabilities, we have developed a framework that describes four characteristics of big data (see Figure 1). If executives are not facing at least three of these four issues, it s unlikely they re confronting a real big data problem, which means their organization s existing capabilities and tools may be sufficient for now. Are you facing a big data problem? Volume Does the problem require that large volumes (currently, from tens of terabytes up through petabytes) be analyzed simultaneously? Some applications demand it. Consider Amazon s recommendation engine, which aggregates and analyzes hundreds of terabytes of shopping cart and click-through data from millions of users to determine which products are related. Combining this information with a user s online behavior generates real-time product recommendations personalized to each Amazon customer. However, online retailers can often deliver fairly accurate recommendations by analyzing just a statistically relevant sample of a large data set. To reduce the size of the data set, most e-commerce sites get their recommendations by noting products that were purchased together. While this approach lacks the personalization available to Amazon, most retailers consider it good enough because it provides many of the same up-sell and cross-sell opportunities at a fraction of the computational cost. Velocity Does the problem require analysis in real time? Wall Street traders need to analyze and execute trades in fractions of a second. They pay millions to gain milli- 1
4 Navigating the big data challenge Figure 1: Data challenges vary by industry Data type (% of structured data) <40% 60% 80% Volume (bytes) PB TB GB 100% O (N) MB h O (0) O (1) O (2) Analytical complexity* (bit complexity) m s ms Velocity** (records/unit time) Retail consumer analytics Utilities optimization Financial services National security *Analytical complexity is measured as the number of operations required to generate an answer from crunching the data **Velocity is defined as the amount of time required to reach an answer for a sample of issues in each issue Source: Bain & Company second advantages by locating their servers as close as possible to the stock exchange, and their firms are developing proprietary big data solutions. For them, big data tools that reduce the processing times from milliseconds to microseconds and enable throughputs from hundreds of thousands to millions of transactions offer enough return to justify the expenditure. because those insights are put to use only in a more traditional time cycle. That may change in the future, but for now a far better investment for most retailers would be to improve their analytical capabilities to make more effective use of existing data and tools, to help them avoid running out of inventory or to group deliveries into fewer shipments. Few organizations need to make decisions that quickly. In brick-and-mortar retail for example, only a handful of companies monitor their supply chain as closely as Wal-Mart does, with its highly publicized RFID (radio frequency identification) tags that track products through their distribution network. But even those organizations still make decisions based on time increments of a few minutes or hours. Spending big money to generate instantaneous insights would be a waste today, Data types Are the data easy to split into meaningful units that can be sorted, evaluated and compared? Traditional analytical software doesn t cope well with unstructured data, including multilingual audio, video, images and text. National security organizations face this challenge every day, as they process petabytes of communication data, security video footage, calls and other 2
5 Navigating the big data challenge raw data. To meet this challenge, governments spend hundreds of millions of dollars to develop and operate big data computer clusters and analytics. Analytical complexity Think of the complexity of a business problem as the number of operations required to transform a set of data into actionable insights. One example is Lexis- Nexis s program to determine whether individuals are related, using a variety of data types, including court records, birth certificates and online records. Financial institutions depend on LexisNexis s system to help prevent identify theft and fraud. By comparison, determining a mobile phone subscriber s monthly bill is comparatively simple, even though the telecom is dealing with massive amounts of call data describing call duration, location, interconnections and roaming. It is relatively simple for the telecom to identify every call (due to the unique mobile number) and aggregate total usage and fees at the end of the month. The analytic process doesn t have to cope with fuzziness, just simple yes or no rules. Where to invest now Companies should focus investments on hiring and developing data scientists who can ask the right questions using their current data systems to determine answers, rather than purchasing new solutions that may be more than they need. Improve your data capabilities Decision makers and their teams will need to develop the capability to ask questions that push the business forward. It won t be easy: Figuring out which questions to ask requires creativity, a deep understanding of the available data and a thorough knowledge of the business. Managers should place a premium on recognizing opportunities in data and on thinking about what could be possible if there were no constraints to getting answers to the questions they might ask. Metrics that show how well teams are using data to meet their goals will become increasingly important. Don t mistake reports for insights The risk of being overloaded by information becomes greater with every terabyte stored. With storage costs continually declining, organizations may be tempted to save everything indiscriminately. But that practice can actually increase the challenge of locating the data that really matters. Managers and teams will need periodically to ask themselves if the data they are storing and reporting is yielding meaningful insights that affect the success of the business. They may even want to stop issuing reports to see who notices and complains which is a proven and effective way to show that data handling was geared toward busywork and reports, overlooking opportunities to create valuable business insights. Restructure your organization to focus on insights from data No organization can hope to gain insights that move their business forward simply by handing their data problem over to the IT department and purchasing a big data solution. The goals of the business will continue to guide the way that companies use data, just as it guides the way they use all their resources. Consider pairing traditional executives with quantitative people 3
6 Navigating the big data challenge who understand and are comfortable working with data. Pairing their complementary skills can help guide teams to decisions that exploit the data opportunity in service of the business s progress. Whether you face a true big data challenge or just have lots of data, the key success factor is building strong capabilities that move the business. Talented decision makers, solid analytical skills and good technology are essential. But the ultimate source of competitive advantage lies in the art of abstracting the potential value within data and turning it into meaningful insights, which should be the goal of any transformation aimed at overcoming your data challenge. Getting better insights from data Bain s approach 2. Benchmark insights and analysis. How do your insights compare with those of competitors and with businesses in other industries that have identified value in their data? 3. Identify and prioritize the opportunities for improving data utilization. How well do you use your data? Do you see opportunities for developing better analytics or asking better questions? Will you need more or different data? Will you need more or faster processing? 4. Identify the resources necessary to realize those opportunities. What new tools, people, analyses, systems and service providers will you need to address the opportunities? 1. Audit your current data systems. Are you capturing and storing the types of information that will help you gain the insights you need? 4
7 Shared Ambition, True Results Bain & Company is the management consulting firm that the world s business leaders come to when they want results. Bain advises clients on strategy, operations, technology, organization, private equity and mergers and acquisitions. We develop practical, customized insights that clients act on and transfer skills that make change stick. Founded in 1973, Bain has 48 offices in 31 countries, and our deep expertise and client roster cross every industry and economic sector. Our clients have outperformed the stock market 4 to 1. What sets us apart We believe a consulting firm should be more than an adviser. So we put ourselves in our clients shoes, selling outcomes, not projects. We align our incentives with our clients by linking our fees to their results and collaborate to unlock the full potential of their business. Our Results Delivery process builds our clients capabilities, and our True North values mean we do the right thing for our clients, people and communities always.
8 Key contacts in Bain s Technology practice: Americas: Europe: Asia-Pacific: Eric Almquist in Boston Chris Brahm in San Francisco Mark Brinda in New York Michael Heric in New York Ron Kermisch in Boston Velu Sinha in Palo Alto Rasmus Wegener in Atlanta Stephen Bertrand in London Arpan Sheth in Mumbai Moonsup Shin in Seoul For more information, visit
Big Data: The organizational challenge If you don t know who (and where) your chief analytics officer is, you may already be behind the curve. By Travis Pearson and Rasmus Wegener Travis Pearson is a partner
fs viewpoint www.pwc.com/fsi 02 15 19 21 27 31 Point of view A deeper dive Competitive intelligence A framework for response How PwC can help Appendix Where have you been all my life? How the financial
For Big Data Analytics There s No Such Thing as Too Big The Compelling Economics and Technology of Big Data Computing March 2012 By: 4syth.com Emerging big data thought leaders Forsyth Communications 2012.
IBM Global Business Services IBM Institute for Business Value Business Analytics and Optimization for the Intelligent Enterprise Business Analytics and Optimization IBM Institute for Business Value IBM
Is Connectivity A Human Right? For almost ten years, Facebook has been on a mission to make the world more open and connected. For us, that means the entire world not just the richest, most developed countries.
Big Data Getting Value from Big Data: Focus on the Opportunities, Not the Obstacles Table of Contents 2 Embark on Your Big Data Journey with Confidence Getting Started, Keeping Moving 3 Big Data Hype Versus
CYBERSECURITY WORKFORCE Getting in Front of the Cybersecurity Talent Crisis http://boozallen.tumblr.com/post/120784624298/ how-to-build-a-cyber-dream-team-when-it-comes-to CONTENTS INTRODUCTION Introduction...
Large project management in oil and gas Large, complex projects demand a coherent reference framework and top-notch project management skills. By Roberto Nava and Tiziano Rivolta Roberto Nava and Tiziano
www.pwc.com PwC Advisory Oracle practice 2012 How to drive innovation and business growth Leveraging emerging technology for sustainable growth 1 Heart of the matter Top growth driver today is innovation
A REPORT BY HARVARD BUSINESS REVIEW ANALYTIC SERVICES The Digital Dividend: First-Mover Advantage Sponsored by SPONSOR S PERSPECTIVE ANTHONY RECINE SENIOR VICE PRESIDENT AND CHIEF MARKETING OFFICER VERIZON
WHITE PAPER Is Your Data Cloud Ready? Of all the questions to be answered before taking advantage of the cloud, this one is the most pertinent; is my data cloud ready? The question is especially important
SAP BusinessObjects Business Intelligence SAP BusinessObjects Business Intelligence 4.0 Solutions Empowering the Real-Time, Mobile, Social, and Global Enterprise SAP BusinessObjects Business Intelligence
Journal of Advertising Research Vol. 54, No. 1, 2014 www.journalofadvertisingresearch.com Lessons Learned from 197 Metrics, 150 Studies, and 12 Essays: A Field Guide to Digital Metrics Stephen D. Rappaport
A REPORT BY HARVARD BUSINESS REVIEW ANALYTIC SERVICES Making the Real-Time Enterprise a Reality Making the Real-Time Enterprise a Reality REAL TIME. ON DEMAND. ALWAYS ON. Such are the expectations of businesses
SAP Solutions for Analytics Big Data Analytics Guide Better technology, more insight for the next generation of business applications Big Data Analytics Guide 2012 Big Data Analytics Guide 2012 Big Data
Convergence of Social, Mobile and Cloud: 7 Steps to Ensure Success June, 2013 Contents Executive Overview...4 Business Innovation & Transformation...5 Roadmap for Social, Mobile and Cloud Solutions...7
A REPORT BY HARVARD BUSINESS REVIEW ANALYTIC SERVICES How the Cloud Looks from the Top: Achieving Competitive Advantage In the Age of Cloud Computing Sponsored by Length of Time Using Cloud Computing Figure
E-PAPER March 2014 Big Data & the Cloud: The Sum Is Greater Than the Parts Learn how to accelerate your move to the cloud and use big data to discover new hidden value for your business and your users.
HOW SAAS CHANGES AN ISV S BUSINESS A GUIDE FOR ISV LEADERS Sponsored by Microsoft Corporation Copyright 2012 Chappell & Associates Contents Understanding the Move to SaaS... 3 Assessing SaaS...3 Benefits
Innovation Roles The People You Need for Successful Innovation A White Paper By Dean Hering Jeffrey Phillips NetCentrics Corporation November 1, 2005 NetCentrics 2005. All rights reserved. 1 Table of Contents
Home Digital Revolution I Combating Evasion & Fraud I Future Compliance Model I Required Capabilities I Our Solution: Trouve I Getting Started I Capgemini & SAS Trouve Our Solution to Combat Tax & welfare
Harvard Business Review March-April 1998 v76 n2 p72(13) Page 1 by Joan Magretta Dell Computer founder Michael Dell is responsible for the success of the $12 billion company. Through an innovative business
Big Success With Big Data Executive Summary Big Success with Big Data 1 2 Big Success with Big Data In organizations that are using big data today, users report overwhelming satisfaction with their results,