Achieving Business Value through Big Data Analytics Philip Russom TDWI Research Director for Data Management October 3, 2012
Sponsor 2
Speakers Philip Russom Research Director, Data Management, TDWI Brian Ng Director, Enterprise Services, HP
Today s Agenda The Need for Business Value from Big Data Definitions of Big Data Analytics Use Cases for Big Data Analytics that deliver Business Value The Future & How to Prepare for It
Background The quantity and diversity of Big Data has been exploding for years Traditional applications grow larger & more numerous every day Older big data sources: RFID, call detail record, machine/robotic data New big data sources: sensors, social media, new Web apps User organizations are starting to achieve business value from big data The consensus today is that Advanced Analytics yields valuable business insights As long as big data is managed well and treated to the right forms of analytics Today we ll look at how Big Data Analytics can deliver business value In your organization is big data considered mostly a problem or mostly an opportunity? 70% 30% Opportunity because it yields detailed analytics for business advantage Problem because it's hard to manage from a technical viewpoint Source TDWI. Survey of 325 respondents, June 2011
Big Data Advanced Analytics Definition of Big Data Analytics It s where advanced analytic techniques operate on big data sets. It s about two things: big data AND advanced analytics. The two have teamed up to leverage big data. The combo turns big data into an opportunity. Big Data isn t new. Advanced Analytics isn t new. Their successful combination is new. Both users and technologies are now more capable of success. The combo is new & technical. But hasn t yet aligned with business. Big Data Analytics
The 3 Vs of Big Data summarize technical properties Business Value should be the 4th V, since this is what IT must deliver. VOLUME VOLUME VELOCITY VARIETY VELOCITY VARIETY BUSINESS VALUE
Defining Advanced Analytics OLAP & its Variants Users have this They ll keep & grow it OLAP won t go away Advanced Analytics Discovery oriented Excels with Big Data Experiencing strong adoption by users Online Analytic Processing (OLAP) It s somewhat rudimentary, but required. Demands multidimensional data modeling, but works well with most EDWs. There are multiple approaches to OLAP. Extreme SQL Uses well-known SQL-based tools & techniques. Relies on long, complex SQL statements. Predictive Analytics Uses data mining and/or statistics to anticipate future events. Multi-Structured Data Analytics Natural language processing (NLP) Search, text analytics, sentiment & social analytic apps Other Analytic Methods Visualization, artificial intelligence Analytic database functions: in-database analytics, inmemory databases, columnar data stores, appliances, etc.
TDWI SURVEY SAYS: Opportunities for Big Data Analytics Anything involving customers benefits from big data analytics better-targeted social-influencer marketing (61%) customer-base segmentation (41%) recognition of sales/market opportunities (38%) BI, in general, benefits from big data analytics more numerous and accurate business insights (45%) understanding business change (30%) better planning and forecasting (29%) identification of root causes of cost (29%) Specific analytics applications are likely beneficiaries detection of fraud (33%), quantification of risks (30%) market sentiment trending (30%) Source TDWI. Survey of 325 respondents, June 2011
USE CASE Exploratory Analytics with Big Data Big Data enables exploratory analytics. Discover patterns and new facts the business didn t know Customer base segments Customer behaviors and their meaning Forms of churn and their root causes Relationships among customers and products Root causes for bottom line costs State of biz today; predict future events
USE CASE Analyze Big Data You ve Hoarded Yes, it s true: Many firms have squirreled away large datasets, because they sensed business value, yet didn t know how to get value out of big data. Finally understand: Web site visitor behavior Products of affinity based on ecommerce shopping carts Product and supply quality based on robotic & QA data from manufacturing Product movement via RFID in retail
USE CASE Big Data Analytics per Industry The type and content of big data can vary by industry, thus have different value propositions per industry: Call detail records (CDRs) in telecommunications RFID in retail, manufacturing, and other product-oriented industries Sensor data from robots in manufacturing, especially automotive and consumer electronics
USE CASE Analytics for Unstructured Big Data Tools based on natural language processing, search, and text analytics (plus new platforms like Hadoop) provide visibility into text-laden business processes: Claims process in insurance Medical records in healthcare Call center and help desk applications in any industry Sentiment analysis in customer-oriented businesses, with both enterprise and social media big data
I love/hate your product! USE CASE Customer Analytics with Social Media Data Customers can influence each other by commenting on brands, reviewing products, reacting to marketing campaigns, and revealing shared interests Predictive analytics to discover patterns, anticipate product/service issues Measuring share of voice and brand reputation Broader input for customer satisfaction Understanding sentiment drivers Voice of the customer analytics Determining marketing effectiveness Identifying new customer segments
USE CASE Big Data for Complete Customer Views Big data can add more granular detail to analytic datasets: Data from all customer touch points Broaden 360-degree views of customers and other entities, from hundreds of attributes to thousands For more detailed and accurate customer base segmentation, direct marketing, and other customer analytics
USE CASE Big Data Can Improve Older Analytics Big data enlarges and improves data samples for older analytic applications: Any analytic technologies that depend on large samples, such as statistics or data mining Fraud detection Risk management Actuarial calculations
USE CASE Analytics with Streaming Big Data Monitoring & Analysis in True Real Time Energy utility, communication network; any grid, service, facility Surveillance, cyber security, situational awareness Fraud detection, risk calc Logistics, truck/rail freight, mobile asset mgt Near Time Review of loan applications submitted online $$$$$ $$$$$ $$$$$ $$$$$ $$$$$
A Look Into the Future of Big Data Analytics 110100110 101110100 100101011 Big data analytics is here to stay It will spread into more apps in more industries, becoming mainstream Big data will be less of a problem Due to advances in storage, clouds, CPUs, memory, databases, analytic tools, etc. Analytics will draw biz value from big data That s why the two have come together New types of analytic apps will appear Old ones will be revamped Big Data Analytics is mostly batch today Will go real time as users/techs mature Analytics is new competency for many shops They will hire & train, plus acquire tools and seek professional services
Recommendations Insist on business value from big data Don t merely hoard it in a cost center that wastes valuable storage & other resources The path to business value is through analytics Go beyond reporting and OLAP into advanced analytics You need discovery analytics, but reporting and OLAP won t go away Embrace the brave new world of big data Data from Web, machine, and social sources Upgrade, extend or distribute your BI/DW tech stack and other software portfolios with technologies for big data and analytics Change is needed to accommodate analytics with big data Give the business what it needs Discovery analytics to understand change, find opportunities Broader, more complete views of customers & other business entities Analytics tailored to your industry and your organization s unique requirements
Achieving business value through big data analytics HP Enterprise Services, Information Management and Analytics, Brian Ng / Oct 2012 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Our point of view Thriving in the age of big data Call Records Sentimen t Risk RFID Sensors Claims Social Fraud We are at a fundamental inflection point in the evolution of information and intelligence. Traditional approaches, architectures and organizations models were not designed for today s complexity. Leadership will be defined by those who excel in information sciences, via innovative solutions, advanced technologies & new 21 talent Copyright models. 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use cases and architecture Copyright Copyright 2012 2012 Hewlett-Packard Development Company, Company, L.P. L.P. The information The information contained contained herein herein is subject is subject to change to change without without notice. notice.
Data capture 1. Unstructured and structured analysis Logical architecture Service management Portfolio management Operations management Event processing Complex event processing Data acquisition Human Language Rich Media Structured data Semistructured data External data Internal data Unstructure d data Rules engine Capture Data transformation Master data Derive metadata and Data index quality Master data mgt Match and Integration combine Aggregation Populate repositories SOA services Repository Raw Data Repository Relational DBMS data warehouse Applications Non-relational DBMS (e.g. Relational HDFS, Hbase,...) DBMS Staging Non-relational Integration Enterprise DBMS DW (content mgt systems) Applications Sentiment, Mark-up & Integrate Data Virtualization Data mart (e.g. OLAP Data cube) marts Olap Real-time cubes analytical RDBMS Analysis and reporting Rules generator Analytical Data mining Data Mart engine SQL analytics engine Reporting Search Dashboards engine Olap Statistical analysis NoSQL Data mining (e.g. Visualization MapReduce) engine Visualization Visualization Analysis Static and OLAP reports Dashboards and alerts Statistical analysis Predictive analysis Governance Data governance Data audit, balance and control 23 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Data capture 2. Machine generated data streams Logical architecture Service management Portfolio management Operations management Event processing Complex Applications event processing Rules engine Data acquisition Sensor Network Structured data Semistructured data External data Internal data Unstructure d data Capture Data transformation Master data Derive metadata and Data index quality Master data mgt Match and Integration combine Aggregation Populate repositories Complex Event Processing SOA Rules services Engine Applications Repository Raw Data Markup, stream, Repository integrate Relational DBMS Data data warehouse Virtualization Non-relational DBMS (e.g. Relational HDFS, Hbase,...) DBMS Staging Non-relational Integration Enterprise DBMS DW (content mgt systems) Data mart (e.g. OLAP Data cube) marts Olap Real-time cubes analytical RDBMS Analysis and reporting Rules generator Analytical Data mining Data Mart engine SQL analytics engine Reporting Search Dashboards engine Olap Statistical analysis NoSQL Data mining (e.g. Visualization MapReduce) engine Visualization Model and Rule development. Static and Real OLAP time reports visualization and analysis Dashboards and alerts Statistical analysis Predictive analysis Governance Data governance Data audit, balance and control 24 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case: Insurance claim fraud Fraud detection Business issue Insurance claim fraud continues to be a major cost Big Data sources Claims form (human language) Contact records (call center logs, audio, email, instant message, video calls) Process Sentiment analysis and meaning-based scoring Input structured result-set into fraud analysis Machine learning for key patterns Business benefit Avoid cost Improve margins Competitive pricing 25 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Data capture Use case: Insurance claim fraud Human language data and analysis Service management Portfolio management Operations management Event processing Complex event processing Data acquisition Claims form contact data Structured data Semistructured data External data Internal data Unstructure d data Rules engine Capture Data transformation Master data Derive metadata and Data index quality Master data mgt Match and Integration combine Aggregation Populate repositories SOA services Raw data repository Repository Claims application Applications Sentiment analyses & Relational integrate DBMS data warehouse Non-relational DBMS (e.g. Relational HDFS, Hbase,...) DBMS Staging Non-relational Integration Enterprise DBMS DW (content mgt systems) Analytical Data data mart Virtualization Data mart (e.g. OLAP Data cube) marts Olap Real-time cubes analytical RDBMS Analysis and reporting Rules generator Data mining engine SQL analytics engine Reporting Search Dashboards engine Olap Statistical analysis NoSQL Data mining (e.g. Visualization MapReduce) engine Visualization Static and OLAP reports Dashboards and alerts Statistical analysis Predictive analysis Governance Data governance Data audit, balance and control 26 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Use case: Operations Optimization Supply Chain Business Issue Under utilized facilities Less effective Supply and Delivery Chains Less accurate R&D Big Data sources Sensors in supply/delivery chains Network sensors (communication, smart grid) Physical sensors (seismic, health, equipment) Process Statistical, Segmentation and Pattern analysis Real time advanced visualization Business Benefit Optimized supply and delivery chain operations Better utilization of facilities 27 Improved R&D results Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Data capture Machine generated data streams Logical architecture Service management Portfolio management Operations management Event processing Process Complex Applications event Rules engine processing Data acquisition Sensor Network Structured data Semistructured data External data Internal data Unstructure d data Capture Data transformation Master data Derive metadata and Data index quality Master data mgt Match and Integration combine Aggregation Populate repositories Complex Event Processing SOA services Applications Repository Raw Data Markup, stream, Repository integrate Relational DBMS Data data warehouse Virtualization Non-relational DBMS (e.g. Relational HDFS, Hbase,...) DBMS Staging Non-relational Integration Enterprise DBMS DW (content mgt systems) Data mart (e.g. OLAP Data cube) marts Olap Real-time cubes analytical RDBMS Analysis and reporting Rules generator Analytical Data mining Data Mart engine SQL analytics engine Reporting Search Dashboards engine Olap Statistical analysis NoSQL Data mining (e.g. Visualization MapReduce) engine Visualization Model and Rule development. Static and Real OLAP time reports visualization and analysis Dashboards and alerts Statistical analysis Predictive analysis Governance Data governance Data audit, balance and control 28 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Information Taxonomy Taxonomy Aggregation HP Changing the Analytics Paradigm Advanced Analytics using Vertica, Autonomy, and Hadoop Information Insight by business analyst Business Users Search Engine, Trends (Market, Consumers, etc) Data Exploration 1 NoSQL Business Objects, Cognos, OBIEE, Microstrategy SQL SAS, R Predictive, Performance, Operations Advanced Analytics 5 2 Meaning Based Computing Operational Data Store Teradata, Oracle, DB2 Analytic Data Store Vertica 4 Taxonomy Aggregation (IDOL) Information Transformation Automated Information Integration Hadoop Unstructured Data Store 3 Structured Transaction Data Device Data Seamless Data Exploration and Analytics Ability explore unstructured information to uncover important attributes, time periods, groups, or areas of information using Non-Sql techniques 1. Conduct Information research using data visualization, trends, and Google like search tools by accessing the Hadoop information repository 2. Leverages a common information Unstructured taxonomy (ontology) that creates Consumer Data business views across all information from all sources 3. Automatically move this data from research to analytics environment 4. Conduct Business Analytics using metrics and KPI s 5. All from real-time information initiated from End User request 29 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Next steps MasterPlan Services Business Value Assessment Information Strategy and Organization Services Business Solutions Social Intelligence Advanced Analytics On Premise Managed Service Cloud Service Strategy Roadmap Design Implement Consume Big Data Experience Transformation Workshop Social Intelligence Workshop EDW OnTrack Workshop BI Implementation Advanced Information Services for HP, SAP and Microsoft 30 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Questions? 32
Contacting Speakers If you have further questions or comments: Philip Russom, TDWI prussom@tdwi.org Brian Ng, HP brian.ng@hp.com