Big Data: Aspirations, Applications, and Analytics
Study overview IBM Institute for Business Value and the Saïd Business School partnered to benchmark global big data activities IBM Institute for Business Value IBM Global Business Services, through the IBM Institute for Business Value, develops fact-based strategies and insights for senior executives around critical public and private sector issues. www.ibm.com/2012bigdatastudy Saïd Business School University of Oxford The Saïd Business School is one of the leading business schools in the UK. The School is establishing a new model for business education by being deeply embedded in the University of Oxford, a world-class university, and tackling some of the challenges the world is encountering. 2
Introduction to big data Defining big data by the opportunities it creates Greater scope of information Integration creates cross-enterprise view External data adds depth to internal data Defining big data New kinds of data and analysis New sources of information generated by pervasive devices Complex analysis simplified through availability of maturing tools Real-time information streaming Digital feeds from sensors, social and syndicated data Instant awareness and accelerated decision making Respondents were asked to choose up to two descriptions about how their organizations view big data from choices above. Choices have been abbreviated, and selections have been normalized to equal 100%. 3
Introduction to big data Big data embodies new data characteristics created by today s digitized marketplace Characteristics of big data 4
Macro findings Three out of four organizations have big data activities underway; and one in four are either in pilot or production Early days of big data era Almost half of all organizations surveyed report active discussions about big data plans Big data has moved out of IT and into business discussions Big data activities Getting underway More than a quarter of organizations have active big data pilots or implementations Tapping into big data is becoming real Acceleration ahead The number of active pilots underway suggests big data implementations will rise exponentially in the next few years Once foundational technologies are installed, use spreads quickly across the organization Respondents were asked to describe the state of big data activities within their organization. Total respondents n = 1061 Totals do not equal 100% due to rounding 5
Key findings Five key findings highlight how organizations are moving forward with big data 1 Customer analytics are driving big data initiatives 2 3 Big data is dependent upon a scalable and extensible information foundation Initial big data efforts are focused on gaining insights from existing and new sources of internal data 4 Big data requires strong analytics capabilities 5 The emerging pattern of big data adoption is focused upon delivering measureable business value 6
Key Finding 1: Customer analytics are driving big data initiatives Improving the customer experience by better understanding behaviors drives almost half of all active big data efforts Customer-centric outcomes Digital connections have enabled customers to be more vocal about expectations and outcomes Integrating data increases the ability to create a complete picture of today s empowered consumer Understanding behavior patterns and preferences provides organizations with new ways to engage customers Other functional objectives The ability to connect data and expand insights for internally focused efforts was significantly less prevalent in current activities Big data objectives Customer-centric outcomes Operational optimization Risk / financial management New business model Employee collaboration Top functional objectives identified by organizations with active big data pilots or implementations. Responses have been weighted and aggregated. 7
Key Finding 2: Big data is dependent upon a scalable and extensible information foundation Big data efforts are based on a solid, flexible information management foundation Solid information foundation Integrated, secure and governed data is a foundational requirement for big data Most organizations that have not started big data efforts lack integrated information stores, security and governance Big data infrastructure Scalable and extensible Scalable storage infrastructures enable larger workloads; adoption levels indicate volume is the first big data priority High-capacity warehouses support the variety of data, a close second priority A significant percentage of organizations are currently piloting Hadoop and NoSQL engines, supporting the notion of exponential growth ahead Respondents with active big data efforts were asked which platform components were either currently in pilot or installed within their organization. 8
Key Finding 3: Initial big data efforts are focused on gaining insights from existing and new sources of internal data Internal sources of data enable organizations to quickly ramp up big data efforts Untapped stores of internal data Size and scope of some internal data, such as detailed transactions and operational log data, have become too large and varied to manage within traditional systems New infrastructure components make them accessible for analysis Some data has been collected, but not analyzed, for years Focus on customer insights Customers influenced by digital experiences often expect information provided to an organization will then be known during future interactions Combining disparate internal sources with advanced analytics creates insights into customer behavior and preferences Transactions Emails Call center interaction records Big data sources Respondents were asked which data sources are currently being collected and analyzed as part of active big data efforts within their organization. 9
Key Finding 4: Big data requires strong analytics capabilities Strong analytics capabilities skills and software are required to create insights and action from big data Strong skills and software foundation Organizations start with a strong core of analytics capabilities, such as query and reporting and data mining, designed to address structured data Big data efforts require advanced data visualization capabilities as datasets are often too large or complex to analyze and interpret with only traditional tools Optimization models enable organizations to find the right balance of integration, efficiency and effectiveness in processes Skills gap spans big data Acquiring and/or developing advanced technical and analytic skills required for big data is a challenge for most organizations with active efforts underway Both hardware and software skills are needed for big data technologies; it s not just a data scientist gap Analytics capabilities Respondents were asked which analytics capabilities were currently available within their organization to analyze big data. 10
Key Finding 5: The emerging pattern of big data adoption is focused upon delivering measureable business value Patterns of organizational behavior are consistent across four stages of big data adoption Big data adoption When segmented into four groups based on current levels of big data activity, respondents showed significant consistency in organizational behaviors Total respondents = 1061 Totals do not equal 100% due to rounding 11
Additional Findings Big data leadership shifts from IT to business as organizations move through the adoption stages CIOs lead early efforts Early stages are driven by CIOs once leadership takes hold to drive exploration CIOs drive the development of the vision, strategy and approach to big data within most organizations Groups of business executives usually guide the transition from strategy to proofs of concept or pilots Business executives drive action Pilot and implementation stages are driven by business executives either a function-specific executive such as CMO or CFO, or by the CEO Later stages are more often centered on a single executive rather than a group; a single driving force who can make things happen is critical Leadership shifts Respondents were asked which executive is most closely aligned with the mandate to use big data within their organization. Box placement reflects the degree to which each executive is dominant in a given stage. Total respondents = 1028 12
Additional Findings Executive desire for quick and precise decisions to keep up with the pace of business drives real-time data needs Reduce the lag Executives are focused on reducing the time between data intake and its availability within business processes This lower latency supports the ability to target customer-centric outcomes, but requires a more resilient infrastructure Acceleration anticipated 40% of executives in the Execute stage expect real-time data to be available within processes The move toward real-time availability will continue to increase as the use of machineto-machine processing and embedded analytics expands Speed to insight Respondents were asked how quickly business users require data to be available for analysis or within processes. Box placement reflects the prevalence of the specific time requirements within each stage. Total respondents = 973 13
Additional Findings Challenges evolve as organizations move through the stages, but the business case is a constant hurdle State the case Findings suggest big data activities are being scrutinized for return on investment A solid business case connects big data technologies to business metrics Getting started The biggest hurdle for those in the early stages is first understanding how to use big data effectively, and then getting management s attention and support Skills become a constraint once organizations start pilots, suggesting the need to focus on skills during planning Data quality and veracity only surface as an obstacle once roll-out begins, again suggesting the need for earlier attention Obstacles to big data Respondents were asked to identify the top obstacles to big data efforts within their organization. Responses were weighted and aggregated. Box placement reflects the degree to which each obstacle is dominant in a given stage. Total respondents = 973 14
There are many Compelling Business Cases, most involving the use of analytics 15
Analytics is broadly defined as the use of data and computation to make smart decisions Data Decision point Possible outcomes Historical Simulated Data instances Reports and queries on data aggregates Predictive models Answers and confidence Feedback and learning Option 1 Option 2 Option 3 Text Video, Images Audio 16
Analytics toolkits will be expanded to support ingestion and interpretation of unstructured data, and enable adaptation and learning New Data Traditional New Methods Adaptive Analysis Continual Analysis Optimization under Uncertainty Optimization Predictive Modeling Simulation Forecasting Alerts Query/Drill Down Ad hoc Reporting Standard Reporting Entity Resolution Relationship, Feature Extraction Annotation and Tokenization Responding to context Responding to local change/feedback Quantifying or mitigating risk Decision complexity, solution speed Causality, probabilistic, confidence levels High fidelity, games, data farming Larger data sets, nonlinear regression Rules/triggers, context sensitive, complex events In memory data, fuzzy search, geo spatial Query by example, user defined reports Real time, visualizations, user interaction People, roles, locations, things Rules, semantic inferencing, matching Automated, crowd sourced Ø Learn In the context of the decision process Ø Decide and Act Ø Understand and Predict Ø Report Ø Collect and Ingest/Interpret Decide what to count; enable accurate counting Extended from: Competing on Analytics, Davenport and Harris, 2007 17
Vestas: Better data analysis capabilities lower costs and improve effectiveness Vestas Wind Systems A/S optimizes capital investments based on 2.5 petabytes of information and big data technologies" Solution Vestas can now help its customers optimize turbine placement and, as a result, turbine performance. Uses a big data solution on a supercomputer -- one of the world s largest to date -- and a modeling solution to harvest insights from an expanded set of factors including both structured and unstructured data Results Business Opportunity Wind turbines are a multimillion dollar investment with a typical lifespan of 20-30 years Placement depends upon a large number of location-dependent factors Vestas has been unable to support data analysis of the very large data sets the company deemed necessary for precision turbine placement and power forecasting due to inadequate infrastructure and reliance on external models Insights lead to improved decisions for wind turbine placement and operations, as well as more accurate power production forecasts Greater business case certainty, quicker results, and increased predictability and reliability Decreased cost to customers per kilowatt hour Reduction by approximately 97 percent from weeks to hours of response time for business user requests Greatly improves the effectiveness of turbine placement 18
Automercados Plaza s: Greater revenue through greater insight Automercados Plaza s uses data analysis and optimization to gain deeper insights into its customers and generate spectacular gains in sales and the bottom line" Solution Automercados Plaza s managers now quickly review daily inventory levels, store sales and cost of goods to see which products are selling and are most profitable, and which promotions are most successful Enables chain limit losses by scheduling price reductions to move perishable items prior to spoilage The solution aids in compliance with government price controls on grocery staples Assists with store location selection Business Opportunity $20M in inventory and more than six terabytes of product and customer data spread across multiple systems and databases Unable to easily assess operations at individual stores using manual processes Needed a comprehensive and timely view of operations that would support and improve decisions about business operations Results Increased annual revenues by 30% Increased annual profits by $7M Decreased time to compile sales tax data by 98% Lowered losses on perishable goods, which comprise approximately 35% of the chain's products Helped executives pinpoint optimal locations for four new grocery stores 19
Santam Insurance: Predictive analytics improve fraud detection and speed up claims processing South Africa s largest short-term insurance company uses predictive analytics to uncover a major insurance fraud syndicate, save millions on fraudulent claims and resolve legitimate claims 70 times faster than before. " Solution Business Opportunity Like most insurers around the world, Santam was losing millions of dollars paying out fraudulent claims every year Expenses were being passed on to the customer in the form of higher premiums and longer waits to settle legitimate claims To improve its bottom line and enhance customer satisfaction, the company needed to detect and stop insurance fraud early in the claims process It also needed to find a way to isolate risky, fraudulent claims so that claims managers could more quickly process lower-risk claims Gained the ability to spot fraud early with an advanced analytics solution that captures data from incoming claims, assesses each claim against identified risk factors and segments claims to five risk categories, separating higher-risk cases from low-risk claims Plans to use propensity modeling to enhance and refine segmentation process as more data becomes Results Identified a major fraud ring less than 30 days after implementation Saved more than $2.5M in payouts to fraudulent customers, and nearly $5M in total repudiations Reduced claims processing time on low-risk claims by nearly 90% Cut operating costs by reducing the number of mobile claims investigations 20
The value of analytics grows by incorporating new sources of data, composing analytic techniques, spanning organizational silos, and enabling iterative, user-driven interaction New format or usage of data Structured or standardized Sources and types of data Low Intent-to-buy trends Sales-based demand forecasting Segmentationbased market impact estimates Scope of decision Multi-modal demand forecasting Price-based demand forecasting (own & competitors) High 21
Future Analytic solutions will apply multiple methods to multiple forms of data Example: Utility Vegetation Management Effective Right of Way vegetation management is critical to streamlined utility operations Traditional Right of Way programs are mainly static-scenario driven on a six year cycle Static and rigid models lead to predominantly reactive operations, which are expensive Focus on narrow corridor widths fails to address severe weather impact A multimodal analytics approach can overcome these shortcomings Structured data (e.g. transmission line maps) and unstructured data (e.g. LIDAR sensor) Advanced modeling to perform a dynamic scenario-driven analysis SENSORS Preprocessor UTILITY DATA Preprocessor MAPS Preprocessor 3-Dimensional Model Recovery Right-of-Way Dynamic Forecasting Model Solution Framework Visualization ELECTRIC TELECOMMUNICATIONS RAIL ROAD OIL WEATHER Preprocessor Schedule Generator 22
Analyzing the data created by internal social interactions transforms individual knowledge into organizational insight Dynamic Recommendation s Using social analytics within the workplace Social Network Building Community Metrics Social Influence Analysis Sentiment Analysis Source: IBM Global Business Services Source: IBM Global Business Services
Eras of Computing Cognitive Systems Era Programmable Systems Era Tabulating Systems Era
Every Generation of Technology Enables Remarkable Outcomes Apollo 11: 2048 words RAM (16-bit word)-> ~4KB 36,864 words ROM Avg. Smart Phone: 256 MB 512 MB Cache 2 GB 64 GB RAM